Wednesday, 25 April 2018

node

1. node concepts
  • http server + request router + request handler
  • use DI everywhere, pass router/handler as parameter to server
  • no 'return' is used, use async function, 'response' object is passed all the way to handler
  • in node, any async function accepts a callback as the last parameter
  • in node, any callback function accepts an error as the first parameter

2. even loop
  • timer: execute timer callbacks
  • pending callbacks: execute I/O callbacks
  • poll: incoming I/O events and callbacks

3. npm config file
  • ~/.npmrc

4. module.export and require
  • module.export return an object
  • require use the object
      module.exports = function(app, db) {};  // file1.js
      require('./app/routes')(app, {});               // file2.js

5. node/nvm commands
  • node -v
  • nvm --version
  • nvm ls-remote
  • nvm install x.x.x
  • nvm uninstall x.x.x
  • nvm use x.x.x
  • nvm use node                                   //switch to latest
  • nvm list
  • nvm install --lts                                //install latest
  • npm install -d                                   //--loglevel info
  • npm install jsonpath --save-dev      //save it in package.json
  • npm install -D jsonpath                   //-D is same as --save-dev
  • npm init                                           //new node app, generate package.json
  • node hello.js

6. node app config file
  • ./config by default
  • can be overwritten: process.env["NODE_CONFIG_DIR"] = __dirname + "/configDir/";

7. commonly lodash functions
  • .find()

8. returning everything as objects of functions

const retryCounter = () => {
    let retryCounter = 0
    const totalRetry = 10 //config.get('errorHandler.retryTotal')

    return {
        retryTotal: () => {
            return totalRetry
        },
        count: () => {
            return retryCounter
        },
        add: () => {
            retryCounter += 1
        }
    }
}

9. print stack trace
  • console.trace("Here I am!")

10. package-lock.json (normally more important than package.json, but)
  • package.json overrule the package-lock.json if package.json has been updated
  • 'npm install' sometimes does not update package-lock.json, better to use 'npm update' 
  • version in package.json
major.minor.patch
1.0.2
~1.0.2 means to install version 1.0.2 or the latest patch version such as 1.0.4
^1.0.2 means to install version 1.0.2 or the latest minor or patch version such as 1.1.0


11. dependencies vs devDependencies
  • dependencies is installed when using 'npm install' or 'npm install $package'
  • devDependencies is installed when using 'npm install' or 'npm install --save-dev $package'
  • dependencies is needed to run a package
  • devDependencies is need to develop a package



reference

Saturday, 7 April 2018

maven

1. lifecycle
  • clean
  • default
  • site
  • packaging of project: if no <packaging> value found, jar is used.

2. phase
  • clean (3)
  • default (21)
  • site (4)
  • 'mvn install' will execute all phases preceding 'install'

3. phase and plugin:goal
  • phase rely on plugin:goal to carryout the task
  • default binding between phase and goal exist:  test | surefire:test
  • 'mvn test' will execute default plugin/goal, i.e. 'mvn surefire:test'
  • plugin/goal can be executed outside of phase, e.g. 'mvn clean dependency:copy-dependencies package'
  • when execute maven you can specify 'phase' or 'goal'
  • if you specify 'phase', it will run all phases up till the specified phase, and it will run all goals attached to each phase
  • if you specify a 'goal', it will run all phases up till the phase for the goal, then it will run that goal
  • mvn-plugins-package-goal

4. mvn profile
  • use profile to customize build for different env
  • use 'mvn -P' option in command line to activate profile
  • profile can also be activated based on OS variable or jdk version

5. mvn command
  • mvn clean //remove target dir
  • mvn test //execute unit test
  • mvn install //install artifact to local repository
  • mvn install -DskipTests //skip test

6. maven surefire and failsafe plugin
  • surefire plugin is used in 'test' phase, it's designed to run unit tests and if any of the tests fail then it will fail the build immediately
  • surefire plugin has one goal: 'test'
  • failsafe plugin is used in 'integration-tests' and 'verify' phase, it's designed to run integration tests, and do not fail the build if there are test failures
  • failsafe plugin has two goals: 'integration-test' runs the integration tests of an application; 'verify' verifies that the integration tests of an application passed

7. parallel/fork
  • run junit in parallel two options

reference

Sunday, 4 March 2018

proxy


1. windows wininet proxy
  • configured in 'LAN settings' (in below order of precedence)
  • 'automatically detect settings'
  • 'use automatic configuration script'
  • 'proxy server'
  • unset $http_proxy                              //unset proxy

2. 'automatically detect settings'
  • If the client is able to automatically detect a proxy script's URL, it will then attempt to download and use that proxy configuration script

3. 'use automatic configuration script'
  • example: http://proxy.contoso.com/proxy.pac

4. 'proxy server'

5. mac system proxy

6. fiddler
  • it use wininet proxy
  • it changes wininet proxy on fiddler start, and restores it on fiddler exit

7. reverse proxy server

8. mock server
  • hoverfly





reference
1. understanding web proxy configuration
2. fiddler in action
3. reverse proxy server
4. how to configure proxy server on mac

Monday, 27 November 2017

browser stack

1. proxy setup
  • webDriverProxy: 'http://606885:summer99@qfwsg-prod.qantas.com.au:8080'

2. local testing
  • test internal server
  • use 'browserstack-local'
  •  exports.bs_local.start({ 'key': exports.config.capabilities['browserstack.key'], 'proxyHost': 'qfwsg-prod.qantas.com.au', 'proxyPort': '8080', 'proxyUser': '606885', 'proxyPass': 'summer99' }, function (error)

3. protractor-multiple-cucumber-html-reporter-plugin + multiple-cucumber-html-reporter
  • connect Protractor, CucumberJS and protractor-cucumber-framework
  • create multiple reports for parallel testing  

4. protractor-cucumber-framework

5. run cucumber/protractor feature files in parallel

6. protractor tips and tricks
  • browser.pause()

7. using local test exe
  • BrowserStackLocal.exe --key 8ceKoJrDNAowyMaFV5py --proxy-host qfwsg-prod.qantas.com.au --proxy-port 8080 --proxy-user 606885 --proxy-pass winter99


reference

Saturday, 4 November 2017

powershell & chocolatey

1. set web proxy

$reg = "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings"
Set-ItemProperty -Path $reg -Name ProxyServer -Value "proxy.example.org:8080"
Set-ItemProperty -Path $reg -Name ProxyEnable -Value 1


2. install chocolatey
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))


3. install and uninstall app
  • cinst jdk8 -y
  • clist -l
  • cuninst packagename

4. execute native command
  • iex

5. execute executable
  • &

6. log console output to file

  • Start-Transcript -path $LogFile -append
  • Stop-Transcript

7. task security

  • run only when user is logged on
  • run whether user is logged on or not (does not work for UI interactive program like autoit)

8. rdp session



ref
1. set window web proxy with powershell
2. powerscript to install dev machine
3. powershell run executables
4. redirect output to file
5. task security when using schtasks
6. rdp session timeout configuration
7. inactive logon windows 2012

Monday, 30 October 2017

protractor with typescript

1. files
  • package.json (npm specific, same as pom.xml in maven)
  • tsconfig.json (typescript specific, typescript project file)
  • vscode cucumber plugin and cucumber-mapper plugin

2. setup
  • npm install -g protractor (global install, can use in command line without specify path)
  • npm install protractor (local install, used by require(), can also use in command line if specify path)
  • npm list <package_name> (list local packages)
  • npm list -g <package_name>  (list globally packages)
  • npm list -g --depth=0 (list globally user installed top level package)
  • npm install <package_name> --save (used for production)
  • npm install <package_name> --save-dev (used for development)
  • both write to package.json -> dependencies
  • npm install (install dependencies in ./package.json)
  • npm run update (install update)

3.tsconfig.json
  • it lists the files belonging to the project as well as compiler options
  • outFile (.ts files are converted to .js and output here, help sourceMap to find)
  • @types, typeRoots and types
  • by default all visible “@types” packages are included in your compilation
  • if typeRoots is specified, only packages under typeRoots will be included
  • if types is specified, only packages listed will be included

4. config.js
  • browser.ignoreSynchronization = true      //this is used for non-angular app, now become selemium js and should use explicit wait
  • browser.ignoreSynchronization = false     //this is used for angular app, and no wait is needed for angular element
  • browser.ignoreSynchronization is depreciated, now should use browser.waitForAngularEnabled
  • SELENIUM_PROMISE_MANAGER: false     //disable control flow
  • when control flow is disabled, can use async/await or promise chain (i.e. .then()) to synchronise command
  • when control flow is enabled, use async/await has conflict with control flow, so need to use promise chain

5. promise & control flow
  • all webdriver js (i.e. protractor) functions are async and return promises
  • webdriver js maintain a queue of pending promises called control flow 

5. debug typescript in vscode
  • "sourceMap": true in tsconfig.json so that you can debug .ts
  • configure 'launch.json' file correctly, it can contain multiple configurations (choose one when debugging)

5. shortcut
  • ctrl+shift+f (search all)
  • f5 (debug)
  • ctrl+f5 (run)
  • ctrl+p (open recent files)

6. await/async
  • async function return a promise
  • await can only be used in async function
  • await make javascript wait until promise resolve

7. webdriver js
  • when promise manager is disabled (SELENIUM_PROMISE_MANAGER=0), each command will execute as soon as it's scheduled
  • when promise manager is enabled, each command will block until the previous command complete
  • turn off control flow (SELENIUM_PROMISE_MANAGER: false) when using async/await
  • setDefaultTimeout(60 * 1000); to change default timeout

8.  npm run or npm test or npm start
  • your current package directory's bin directory is placed at the front of your path

9. npm ls selenium-webdriver

10. node js
  • single threaded
  • concurrency model based on event loop and registered callback

11. style guide
  • make test independent at least at file level

12. debug protractor in vscode
  • .vscode/launch.json

"name": "Launch",
"type": "node",
"request": "launch",
"program": "${workspaceRoot}/node_modules/protractor/bin/protractor",
"stopOnEntry": false,
"args": ["${workspaceRoot}/protractor.conf.js"],


reference

Sunday, 22 October 2017

aws

1. aws cli credential (also see item 9)
  • '~/.aws/config' file, or
    '~/.aws/credentials' file, or
    'AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY' env variable, or
    ec2 instance profile credentials
  • 'credentials' stores access keys
  • 'config' stores other configuration like region and output format
  • no conflict between 'credentials' and 'config', aws cli or java sdk will look at both file
  • when using aws cli, set 'AWS_PROFILE' env variable or use --profile flag in cli command if profile is not default
  • when using java sdk, use ProfileCredentialsProvider if profile is not default

2. iam user/group/role/policy/profile
  • user is used to sign in  (aws iam list-users)
  • group is collection of users
  • role is collection of policies (user can have multiple roles)
  • policy=permission
  • profile is a settings file

3. arn 
  • amazon resource name
  • arn:partition:service:region:account:resource        //format
  • arn:aws:iam::998355367879:user/test-user            //example: user

4. kinesis
  • streaming data service
  • consumer: firehose, data analytics etc
  • pip3 install kines
  • kines walk streamotion-gam-datalake-nonprod  000000000120 -l -f    //000000000120 is one of the shards id. shard == partition 

5. commands
  • aws configure list-profiles         //show all profiles
  • aws configure list                      //show current profile
  • aws sts get-caller-identity         //current user, role

6. profile
  • [default] is used when running cmd without --profile param
  • export AWS_PROFILE=user1                  //use the named profile

7. cli settings precedence (which override which)
  • command line option                    //--profile
  • environment variable                    //export 
  • credentials file
  • config file

8. ec2 key pair
  • when ec2 instance boost, public key is stored in instance
  • when connecting to ec2 instance, you must specify private key

9. set credentials when use java sdk or aws cli
  • option 1: use '~/.aws/credentials' file to set up credential (not 'config' file)
[default]
role_arn = arn:aws:iam::111111111111:role/group-xxxxx-xxxxx
source_profile = okta
region = ap-southeast-2

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

  • option 2: use aws environment variable to set up credentials
export AWS_ACCESS_KEY_ID=your_access_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_access_key
export AWS_REGION=your_aws_region
  • aws env variable take precedence over ~/.aws/credentials, but in general, it's best practice to store credentials in the ~/.aws/credentials file, and load the credentials in your code using the AWSCredentialsProvider class
AWSCredentialsProvider credentialsProvider = new ProfileCredentialsProvider();


10. aws lambda vs ec2
  • ec2: need to provision container, orchestration, scaling
  • lambda: taken care of by aws

  • ec2: need to take care of security layer access
  • lambda: taken care of by aws

  • ec2: combined with ALB, has no timeout limit
  • lambda: combined with api gateway, has timeout limit of 15min for lambda and 30 sec for gateway

  • ec2: need to config auto scaling group
  • lambda: taken care of by aws

  • ec2: always available
  • lambda: on demand

  • ec2: cold start not needed unless for a new container
  • lambda: cold start needed

11. api gateway
  • default max integration timeout limit 30 sec

12. skill used at work
  • management console, cli, sdk
  • s3, secret manager, kinesis, sqs, nosql
  • ec2 - elastic compute cloud (vm)
  • ecs - elastic container service (docker)

reference