namaste

This is the namaste repository - a backend, AMQP-based, event-driven framework written in PHP 7.1.

Creation Date: Jun 7, 2017

Last Updated: Oct 20, 2020

Author: mike@givingassistant.org

Reference Link for MD Syntax:

• https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md

Assumptions

• you've already installed git and configured the git globals
• you've pulled the source code from the bit-bucket repository
• you're installing on Ubuntu 18.04 LTS or a cloud-compatible release version

Installation Notes:

Once you've pulled source code, and upgraded to PHP 7.1 (below), you'll need to make a few changes to the namaste environment. From the root directory in the Namaste source, enter the following commands:

• mkdir logs pids
• chmod 1777 logs pids

Some commands to run for setting up the Namaste environment:

• Install Xdebugger 2.7.2 for PHP7.0: https://xdebug.org/wizard.php
  • do not forget to also configure this in PHPStorm when creating the repos!
• cd into the ./lib directory to install composer:
  • curl -sS https://getcomposer.org/installer -o composer-setup.php
  • validate the composer download:
    • php -r "if (hash_file('SHA384', 'composer-setup.php') === '669656bab3166a7aff8a7506b8cb2d1c292f042046c5a994c43155c0be6190fa0355160742ab2e1c88d40d5be660b410') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
    • Output: "Installer verified"
  • Install composer:
    • sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
    • on completion, composer is available globally.. cd into the ./lib directory to install packages needed for the project.
    • move the composer-setup.php file into the ./lib directory
    • run: php composer-setup-php -- this will create the composer.phar file in ./lib
    • visit: https://packagist.org/packages/theseer/phpdox to ensure that the packages listed are version compatible and at the lates revs
    • php composer.phar self-update -- updates composer package to the latest version.
      • Note: composer.phar is NOT checked-in to source -- it is your responsibility to keep it current!
    • php composer.phar update -- installs libraries and dependencies into the lib directory
      • Note: the vendor sub-directory, where composer stores it's files, is not checked into source. Again, it is your responsibility to ensure it is kept current.
    • on completion, you should remove composer-setup.php
• Oracle Java 8
  • Install the Oracle Java8 SDK (required for DynamoDB)
    • instructions: http://tipsonubuntu.com/2016/07/31/install-oracle-java-8-9-ubuntu-16-04-linux-mint-18/
• Download and install AWS DynamoDB
  • instructions: https://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/GettingStarted.Download.html
    • note: the Ddb sdk is installed via composer
• Install memcached:
  • apt install memcached php-memcached

PHP 7.4

Don't forget to enable the Apache mod for PHP 7.4!

Here's the current list of PHP 7.4 packages I currently have installed:

mshallop@clydesdale:~$ apt list "*php*" | grep -i installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libapache2-mod-php7.2/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
libapache2-mod-php7.4/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic]
php-apcu/bionic,now 5.1.18+4.0.11-1+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php-apcu-bc/bionic,now 1.0.5-1+ubuntu18.04.1+deb.sury.org+20191129 amd64 [installed,automatic]
php-common/bionic,bionic,now 2:72+ubuntu18.04.1+deb.sury.org+1 all [installed,automatic]
php-igbinary/bionic,now 3.1.0+2.0.8-2+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic]
php-memcached/bionic,now 3.1.4+2.2.0-1+ubuntu18.04.1+deb.sury.org+20191129 amd64 [installed]
php-mongodb/bionic,now 1.6.1-4+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php-msgpack/bionic,now 2.0.3+0.5.7-2+ubuntu18.04.1+deb.sury.org+20191129 amd64 [installed,automatic]
php-pear/bionic,bionic,now 1:1.10.8+submodules+notgz-1+ubuntu18.04.1+deb.sury.org+1 all [installed]
php-php-gettext/bionic,bionic,now 1.0.12-0.1 all [installed,automatic]
php-phpseclib/bionic,bionic,now 2.0.9-1 all [installed,automatic]
php-tcpdf/bionic,bionic,now 6.2.13+dfsg-1ubuntu1 all [installed,automatic]
php7.2/bionic,bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 all [installed]
php7.2-bcmath/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-bz2/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic]
php7.2-cli/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-common/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-curl/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-gd/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-gmp/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic]
php7.2-imap/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-json/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-mbstring/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-mysql/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-opcache/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-readline/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-xml/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.2-zip/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic]
php7.4/bionic,bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 all [installed]
php7.4-bcmath/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-bz2/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-cli/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic]
php7.4-common/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-curl/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-dev/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-gd/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-gmp/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-intl/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-json/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic]
php7.4-mbstring/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-mysql/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-opcache/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic]
php7.4-readline/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic]
php7.4-xml/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
php7.4-zip/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed]
phpmyadmin/bionic,bionic,now 4:4.6.6-5 all [installed]
pkg-php-tools/bionic,bionic,now 1.35ubuntu1 all [installed,automatic]

To install these tools in a single command:

$ apt install php-memcached php-mongodb php-pear php7.4-bcmath php7.4-bz2 php7.4-curl php7.4-imap uw-mailutils php7.4-mbstring php7.4-mysql php-apcu php7.4-gd libgd-tools php7.4-xml php7.4-zip php7.4-dev php7.4-intl phpmyadmin

Finally, install the pecl libraries for mongoDB and the XDebugger:

$ pecl install mongodb xdebug

RabbitMQ

Install RabbitMQ:

RabbitMQ can fail to install based on your current OS version. As of this writing, we're using the following versions:

• RabbitMQ 3.7.7
• Erlang 20.2.2

You can go to this site: https://www.rabbitmq.com/download.html and choose the appropriate package to download (Linux/Debian for Ubuntu) and then use your software installer to install the deb package. This will install the compatible versions of both RabbitMQ and Erlang.

Or, you can install manually...

$ apt install rabbitmq-server

Once installed, next, install the RMQ management console:

$ rabbitmq-plugins enable rabbitmq_management

Test/confirm that you can access the console via: http://localhost:15672

Generally speaking, I've found user-management via the gui to be pretty janky - you'll be better off using the command line:

$ rabbitmqctl add_user namaste 4YyrWuKH

This creates the namaste user and sets the password.

$ rabbitmqctl set_user_tags namaste administrator

This sets the namaste account to administrator level.

$ rabbitmqctl add_vhost mdev

Adds the vhost "mdev" to the configuration. (ymmv)

$ rabbitmqctl set_permissions -p mdev namaste ".*" ".*" ".*"

This command grants the namaste user full access to the mdev vhost

Helpful Links:

• https://www.rabbitmq.com/man/rabbitmqctl.1.man.html

• https://github.com/rabbitmq/rabbitmq-server/blob/stable/docs/rabbitmq.config.example

Upgrade Erlang

You do not need to do this step if you downloaded and installed from the deb package!

The default version of Erlang, installed with RabbitMQ, under Ubuntu, is sub-optimal for node.js clients. If you're using node.js to connect to RabbitMQ, you should be at Erlang v. 19.x or later.

To upgrade Erlang, follow these steps:

• lsb_release -c

returns the codename of your installation -- for Ubuntu 16.04 or later, this should come back as "xenial".

Add the following line to /etc/apt/sources.list:

• deb http://packages.erlang-solutions.com/ubuntu xenial trusty contrib

(Change "xenial" to the appropriate codename)

Execute the following commands:

• wget http://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc

• sudo apt-key add erlang_solutions.asc

Then, update the system:

• apt update

• apt upgrade

• apt dist-upgrade

Restart RabbitMQ:

• service rabbitmq-server restart

And check the console page for the server - the upper-right corner shows the versions of RabbitMQ and Erlang. Your version of Erlang should now be at 19.x or later.

(As of this writing, the Erlang version is 20.0 for Ubuntu 16.04)

php.ini tweaks:

For most resources, (mongoDB, RabbitMQ, etc.), I like to re-route the logging out put to a single repository. Normally, I use /home/logs however ymmv.

error_log = /home/logs/php_errors.log

Local Variables:

[xdebug]  
xdebug.remote=1  
xdebug.remote_port=9000  
xdebug.remote_autostart=0  
xdebug.var_display_max_depth=10  
xdebug.cli_color=1

mongodb

For mongo, you will need to install the pecl libraries containing the mongodb drivers for PHP 7.1.

pecl install mongodb

Add the following line to the file: /etc/php/7.1/cli/php.ini

[mongodb]
extension=mongodb.so

For Apache2 support, you will need to install the mongodb drivers and restart Apache:

apt install php-mongodb
...
systemctl restart apache2

Apache2 support is necessary for the Namaste browser utilities.

**User RBAC (Role Based Access Controls) **

You should create a minimum of two user account for Namaste mongo when using version 3.6 or later.

The first user should be a root user.

> use admin
> db.createUser({ user: "myRootUser", pwd: "myRootPassword", roles : [ { role : "root", db: "admin" } ] } )

Once the user is created, then you can test your authentication (while still using the admin db):

> db.auth("myRootUser", "myRootPassword")
1

A value of 1 will be returned if the information is correct. If not, a 0 is returned.

Next, create the user accounts. As of this writing, there are two Namaste mongo collections, one on the admin service, and one in Namaste proper. These are discrete databases and you have the option/ability to create different users for both.

Both are defined in the XML configuration

                <user>gaOwner</user>
                <password>zPd7B6^Y</password>
                <useAuth>1</useAuth>
                <authSource>givva_namaste</authSource>

Note that the authSource tag uses the name of the Namaste database sans the environment. (Defined in id -> envName) This will be prepended to the database name prior to making a resource connection request.

To create users for the Namaste and Admin databases:

> use development_givva_namaste
> db.createUser( { user : "gaOwner", pwd: "zPd7B6^Y", roles: [ { role : "dbOwner", db: "development_givva_namaste" } ] } )
> use development_givva_namaste_admin
> db.createUser( { user : "gaOwner", pwd: "zPd7B6^Y", roles: [ { role : "dbOwner", db: "development_givva_namaste_admin" } ] } )

MariaDB

As root:

apt install mariadb-server mariadb-client
mysql_secure_installation

Running mysql_secure_installation will prompt you for the root password - if this is a new, fresh, install, you should just press enter.

Create the gaAdmin account -- if you change the name of the account user, make sure you update the env.xml configuration file also!

CREATE USER 'gaAdmin'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'gaAdmin'@'%';
FLUSH PRIVILEGES;

The database name is derived from the XML file setting under application --> id --> envName which is prepended to the string "givva".

In order to execute Namaste db creation scripts, the gaAdmin user account must have all privileges, globally, minus the GRANT option.

Once the gaAdmin (or whatever you've named your user) has been created in mySQL, create the databases:

CREATE DATABASE development_givva_namaste;
CREATE DATABASE development_givva_namaste_warehouse;

Note that your database name may be different depending on the environment to which you're deploying Namaste. The supported environments are [development | qa | staging | production]. This naming conventions allows us to run databases for different, non-production, environments on the same database instance.

Don't forget to run the ./utilities/mysqlConfig.php program to create your tables, views, and database-objects!

Tips...

If the database fails to start, and you've changed the location of your datafiles to the /home directory, check status using the following command:

$ systemctl status mariadb

And, within the ouput of that you command, you see the following:

... [Warning] Can't create test file /home/data/mysql/master.lower-test

Then you'll need to edit the file located in: /lib/systemd/system/mariadb.service and change the following line:

ProtectHome=true

to this:

ProtectHome=false

Next, reload the system configuration files with this command:

$ systemctl daemon-reload
$ systemctl restart mariadb

This will allow you to start-up mariadb using the /home directory for data storage.

mySQL Timeouts

When there is no broker activity requiring interaction with mySQL as a service, the connection in Namaste to the mysql resource may eventually time-out. When this happens, you will need to restart the Namaste brokers (framework).

The timeout interval can be mitigated by increasing the timeout value in mySQL/MariaDB to a longer interval value. You can increase the timeout value by adding the following lines to the mariaDB configuration file:

# increase timeout values for broker connections
wait_timeout		= 86400
interactive_timeout 	= 86400

Note that you will have to make these changes for every node in the mariaDB cluster.

Make sure you restart the mariaDB service in order for these changes to take effect.

Apache Virtual Host for Namaste:

You should edit and add this v-host configuration to your Apache configuration (/etc/apache2/sites-available/namaste.conf) so that you can access Namaste tools and utilities over HTTP.

# Namaste Virtual Host Configuration
<VirtualHost *:80>
        ServerName namaste
        ServerAlias namaste
        ServerAdmin mike@givingassistant.org
        DocumentRoot /home/mshallop/code/php/namaste
        DirectoryIndex  index.php
        <Directory /home/mshallop/code/php/namaste>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride All
                Satisfy Any
                Require all granted
        </Directory>
        LogFormat       "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
        LogFormat       "%h %l %u %t \"%r\" %>s %b" common
        LogFormat       "%{Referer}i -> %U" referer
        LogFormat       "%{User-agent}i" agent
        CustomLog       /home/logs/namaste_log common
        ErrorLog        /home/logs/namaste_error.log
        CustomLog      /home/logs/namaste_access.log combined
 
        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn
        ServerSignature Off
</VirtualHost>

Local Tools

Namaste comes with a couple of brilliantly-written apps for debugging. When you've installed and aliased Namaste locally as a HTTP virtual host, you can access these tools from your host alias (in this case: namaste):

• cachepeak: memcache reader
  • http://namaste/utilities/cashpeak.php
• dumper: reads namaste log and metrics log
  • http://namaste/utilities/dumper.php

PHPUnit Testing

If you're using PHPStorm, you will want to make a configuration change for PHP Unit testing.

Under "Languages and Frameworks" --> PHP --> Test Frameworks

Select the "Path to phpunit.phar" option and download phpunit.phar -- for my installation, I installed the phar file into:

./lib/phpunit (not checked-in to git)

And was able to successfully run unit tests thereafter.

Other Recommended Tools and Packages:

GitKraken - The legendary GIT GUI client for all platforms:

• https://www.gitkraken.com/

Robomongo - (Linux) Mongo GUI Tool:

• https://robomongo.org/download

Adminer -- Database management in a single PHP File:

• https://www.adminer.org/

Version 2.0 Features

DynamoDB

For this framework, we're using DynamoDB - an AWS nosql storage schema. Review the namaste-readme.txt file in the Ddb directory for help on connecting after you've read the online documentation on AWS.

Key Points:

• DynamoDB is a NoSQL database, and is schemaless, which means that, other than the primary key attributes, you do not need to define any attributes or data types at table creation time.

• Your applications must encode binary values in base64-encoded format before sending them to DynamoDB. Upon receipt of these values, DynamoDB decodes the data into an unsigned byte array and uses that as the length of the binary attribute.

• When your application writes data to a DynamoDB table and receives an HTTP 200 response (OK), all copies of the data are updated. The data will eventually be consistent across all storage locations, usually within one second or less. DynamoDB uses eventually consistent reads, unless you specify otherwise.

• When you create a table, you specify how much provisioned throughput capacity you want to reserve for reads and writes. DynamoDB will reserve the necessary resources to meet your throughput needs while ensuring consistent, low-latency performance. You can also change your provisioned throughput settings, increasing or decreasing capacity as needed.

• DynamoDB is not supported under the AWS Resource API - meaning that you cannot use the object-oriented abstraction layer for coding.

Helpful Links:

• https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.API.html

• https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html

• https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/BestPractices.html

• https://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/GettingStarted.PHP.01.html

• https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html

• https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html

• https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html#Query.Pagination

• https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html

Command Line Examples:

If you want to run aws commands from the command line, you should first run the command:

• aws configure

and follow the prompts to save your key-id and access-key to a local .ini file.

list existing tables:

• aws dynamodb list-tables --endpoint-url http://localhost:8000

delete an existing table:

• aws dynamodb delete-table --table-name development_gaLogs --endpoint-url http://localhost:8000

list a table's attributes:

• aws dynamodb describe-table --table development_gaLogs_log --endpoint-url http://localhost:8000

dump a table using scan:

• aws dynamodb scan --table development_gaLogs_log --endpoint-url http://localhost:8000