botbotbot 's blog

OSX Dev Tools

Package Manager

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew tap caskroom/cask
$ brew tap caskroom/fonts
$ brew tap beeftornado/rmtree && brew install brew-rmtree
$ brew rmtree <package>

Developer Tools

# this is too old version
$ brew cask install iterm2

# try night build that lastest version
$ brew tap caskroom/versions
$ brew cask install iterm2-nightly
$ curl -L https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh | sh
$ brew cask install dash

Window Manager

$ brew cask install spectacle

Etc

$ brew cask install flux
$ brew cask install slack

Jenkins CI

Jenkins-CI - Extendable CI Server

Install Jenkins::

Debian

# Don't use only apt-get install jenkins that old version
$ wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
$ sudo vim /etc/apt/sources.list
# add the following entry
# deb http://pkg.jenkins-ci.org/debian binary/
$ sudo apt-get update
$ sudo apt-get install jenkins

OSX

brew update
brew install jenkins

Allow others to access jenkins

Original From Setting up an Apache Proxy for port 80 -> 8080 but it use old version of apache2

$ sudo apt-get install apache2
$ sudo a2enmod proxy
$ sudo a2enmod proxy_http
$ sudo a2dissite 000-default.conf

Create “jenfins.conf” in /etc/apache2/sites-available

<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  ServerName ci.company.com
  ServerAlias ci
  ProxyRequests Off
  <Proxy *>
		Order deny,Alloww
		Allow from all
  </Proxy>
  ProxyPreserveHost on
  ProxyPass / http://Proxylocalhost:8080/
</VirtualHost>
$ sudo a2ensite jenkins
$ sudo service apache2 restart

Jenkins Job Workspace::

  1. look jenkins home directory in Manage Jenkins > Configure System > Home directory
  2. In Home directory clicks Advanced... Buttons.
  3. look Workspace Root Directory. Normally it is workspace.
  4. your job workspace is < jenkins home directory > /jobs/< your jobs > / < workspace >

Jenkins backup and Import Configure::

Backup Job Configure

http://yourjenkinsserver/job/<project name>/config.xml

Import Job Configure

  1. look jenkins home directory in Manage Jenkins > Configure System > Home directory
  2. go to < jenkins home directory >/jobs
  3. go to < your jobs directory >
  4. replace your backup config.xml with config.xml
  5. reload jenkins with

http://yourjenkinsserver/reload

Jenkins Trigger via git hook::

Setup Configure

  1. Install Git Plugin
  2. In Job Configure > Source Code Management > Select Git
  3. add your git repository url to Repository URL
  4. Select Poll SCM in Build Triggers and you can leave Schedule as blank
  5. Save Configure

Git Hook to trigger Jenkins

  1. If you use linux you have to install curl. That was installed osx.

    $ sudo apt-get install curl

  2. You can trigger jenkins with this url

    http://yourjenkinsserver/git/notifyCommit?url=<URL of the Git repository>

  3. You can trigger jenkins after git push by

    # git don't have post-push hook
    $ git push && curl http://yourjenkinsserver/git/notifyCommit?url=$(git remote -v | grep -m 1 -oh "git@.*git")
    
  4. You can use git alias to make short cmd by edit .git/config in your git repository

     [alias]
       trig-jenkins = !curl http://yourjenkinsservergit/notifyCommit?url=$(git remote -v | grep -m 1 -oh "git@.*git")
       push-wt = !git push && git trig-jenkins
    

call your alias by

   $ git trig-jenkins
   $ git push-wt

Jenkins with Python::

Setup

  1. Install ShiningPanda Plugin
  2. In Manage Jenkins > Configure System > Python clicks “Python installations” button.
  3. Config your python environments
  4. Save

Configure Job to run with python

  1. In Job Configure > Build > Add build step > {Python Builder, VirtualEnv Builder}
  2. Select your Python version
  3. Nature select shell
  4. In command add your run script

    # example
    export LANG=en_US.UTF-8
    python setup.py develop
    nosetests --with-coverage --cover-erase --cover-package=core,bin --cover-html --with-xunit
    
  5. (Optional) Publish Test Result Report

    5.1 Use nosetests to generate xml result in xunit format

    $ nosetests --with-xunit  # nosetests.xml is default name of xml
    

    5.2 Add post-build action > Publish JUnit test result report
    5.3 Add “nosetests.xml” in Test report XMLs

  6. (Optional) Publish coverage.py HTML reports

    6.1 Use nosetests to run coverage.py

    $ nosetests --with-coveage --cover-html
    

    6.2 Add post-build action > Publish coverage.py HTML reports

  7. (Options) Publish your product

    7.1 Add post-build action > Archive the artifacts
    7.2 Add your file name

Start/Stop Jenkins that installed via Homebrew

# launchd start jenkins at login:
$ ln -sfv /usr/local/opt/jenkins/*.plist ~/Library/LaunchAgents
# start jenkins
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.jenkins.plist
# stop jenkins:
$ launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.jenkins.plist

Jekins Reset Password

  1. Go to Jenkins Home Path
    (debian) /var/lib/jenkins
  2. Edit config.xml
  3. Set < userSecurity> tag to false
  4. Delete < authorizationStrategy> and < securityRealm> tags
  5. Restart Jenkins
$ sudo service jenkins restart

SSH Key on Jenkins

$ sudo su - jenkins
$ ssh-keygen -t rsa -C "your_email@example.com"
$ cat ~/.ssh/id_rsa.pub

Jekins git pulling timeout after 10 min

  1. Edit job configuration, In git plugion section
  2. Click Add > Advanced clone behaviours > Set Timeout

Jenkins running parallel scripts

# wokr in any shell
$ script1.sh &   # run in background (sub-shell)
$ script2.sh     # run in foreground (shell)

Jenkins with Docker.io

  1. Move fast and don’t break things! Testing with Jenkins, Ansible and Docker
  2. jenkins-docker-sample
  3. Use Docker + Jenkins to run GitHub tests
# add jenkins to docker group
$ sudo gpasswd -a jenkins docker # or sudo adduser jenkins docker
$ sudo service docker.io restart
$ sudo service jenkins restart

Jenkins Plugins - Install via Terminal::

  1. Git Plugin
  2. ShiningPanda Plugin - Python support to Jenkkins
  3. Green Balls
  4. Build Pipeline Plugin
  5. Build Monitor Plugin
  6. Jenkins Sounds plugin
  7. Robot Framework Plugin
  8. Sonar plugin
  9. Violations - generates reports static code violation detectors
  10. Publish over SSH
  11. Jenkins Job DSL

Jenkins References::

  1. Installing Jenkins on Ubuntu
  2. Jenkins Debian packages
  3. Jenkins and Python
  4. Jenkins :: สรุป plugin ยอดฮิตประจำเดือนพฤษภาคม ปี 2557
  5. Jenkins Setup on Os X server
  6. Jenkins and Python
  7. Integration of pylint into jenkins
  8. Running Jenkins in Docker Containers

Travis CI

Travis CI - Test and Deploy Your Code with Confidence

Travis with Docker

Travis support docker but not fully permission some dependencies that try to use OS command ex. kill can make your test error or failed.

Travis with Python

Nightli.es Travis Night Build

Third-party web service that use Travis API to automatic build every 24 hour after last build

References

  1. Travis CI for Complete Beginners
  2. Faster Builds with Container-Based Infrastructure and Docker
  3. Travis Custom Deployment

Drone.io CI

Drone - Continuous Integration platform built on Docker, written in Go - better than Drone.io

Drone.io based on Drone - Simple CI Server

Private Configuration:

Can run shell command

  1. upload your configuration into gist with secret
  2. get url of your raw configuration
  3. add your configuration url in “Environment Variable”

    # Example
    CONFIG_URL="<your_configuration_url>"
    
  4. in “Commands” download your configurations via wget without display result

    #!/bin/sh
    wget $CONFIG_URL -O <config_name> &> /dev/null
    

Keep Variables in Environment

# Example
#!/bin/sh
export CLIENT_ID=<your id>
export TAGS=<your tags>
#!/usr/bin/env python
import os
client_id = os.environ.get('CLIENT_ID',<default value>)
tags = os.environ.get('TAGS',<default value>)

Dynamic Solution

#!/usr/bin/env python
import os

# Configuration from Environment Variables if can't load config.ini
# you can make you own configuration from config.ini.sample

if os.path.isfile("config.ini"):
    import configparser
    config = configparser.ConfigParser()
    config.read('config.ini')
    client_id = config.get('DEFAULT', 'client_id')
else:
    client_id = os.environ.get('CLIENT_ID', <default value>)

References:

  1. Drone.io Quickstart
  2. Drone.io Building Python Projects
  3. Hosting Your Open Source Project on Heroku with Private Configuration
  4. How to hide connection string, user name, pw when using source control?

Python Line API

Line API requirement - Line API Documentation

Install Thrift on Ubuntu

$ apt-get install -y automake libtool flex bison pkg-config g++ libssl-dev make libqt4-dev git debhelper cmake
$ cd /tmp
$ curl http://archive.apache.org/dist/thrift/0.9.1/thrift-0.9.1.tar.gz | tar zx
$ cd /tmp/thrift-0.9.1
$ ./configure
$ make && make install
$ rm -rf /tmp/thrift-0.9.1

Fix login with email and password failed

# installl line and remove first that tricky don't know why
$ pip install line && pip uninstall -y line
$ git clone https://github.com/carpedm20/LINE.git
$ cd LINE/line
$ wget https://gist.github.com/windows98SE/b739038218b6fe4d423f/raw/5f68cf3d9a2a88576b739810a6bd6fcaa0c5e940/api.py.patch
$ patch api.py < api.py.patch
$ python config.py
$ python setup.py install

Sample Bot

Line API Patched Dockerfile on Ubuntu

Line send message by LineID

References:

  1. Python Line API Repository
  2. Line API Documentation
  3. StepHack.Com : มาเขียน บอท line ด้วย Python กันเถอะ
  4. How to use LINE API in python ( lt’s work)
  5. Line reverse engineering