Monday, November 6, 2017

RBTools and Review Board's Web API

In Review Board RBTools Example I developed a simple client using RBTools. Here I explore another approach using the URI templates provided by Review Board's Web API.

The Review Board Web API embeds a lot of functionality. It enables client development, the documentation describes how to ensure forward compatibility and legacy APIs are managed. It's a nice piece of documentation for a rich API.

The entire API can be obtained using the Root Resource List. This resource insulates clients from URI changes. The URI Templates identify how to obtain specific resources.  To obtain a resource published by the API use the URI template for that resource and fill in the variables.

For example, to obtain a diff in a review request use the URI template for a diff [1].

https://reviews.reviewboard.org/api/review-requests/{review_request_id}/diffs/{diff_revision}/

Obtain the URI, review_request_id and diff_revision as follows:

Although crude, this client uses the Web API in a forward compatible manner. It also provides insight on the relationship between the URI templates and the values to populate to access a resource.

A good next step might be to develop a discovery mechanism using the URI templates and the linked resources so that tools can be created using the Web API.

[1] URI Template.

Sunday, October 8, 2017

Individuals and interactions over processes and tools

Recently, I've had the opportunity to reflect on the value of individuals and interactions over processes and tools, part of the Agile Manifesto.

The team has created a working agreement for design reviews. This agreement requires the involvement of two people and the designer. The rationale for this approach is that it promotes better design and knowledge sharing amongst team members. Interactions are critical to good design because different perspectives can identify opportunities and alternative approaches.

The team struggled with design reviews. To their credit, we are almost at the point where design review are a regular practice. Unfortunately, a few developers have challenges adhering to the working agreement because they feel that it's an impediment for some activities.

In one example, a developer gave a student a design to implement and then went on vacation. Unfortunately, the implementation failed code review when questions arose about the implementation and the student was unable to explain the design rationale. In another example, a developer took liberties in an implementation that broke a best practice.

In the first case, the team was able to improve the resulting design significantly over what the original developer created. In the second case, the best practice got discussed and differences in approach got worked out.

In each example, the developers who avoided the working agreement introduced other challenges that they didn't anticipate. The redesign and implementation in the first example cost an additional week plus the vacation time. The second example created a knowledge void.

The rationale given by each developer that motivated these examples was that they knew what they were doing. In my opinion, they failed to recognized the benefit introduced through the intent of the working agreement: ensure the appropriate interactions occurred.

When the first example went through the sprint retrospective, we ended up with a simple result: require the interaction, use the stand up to create awareness of the design intent and invite participation.

Friday, September 15, 2017

Safer Packer Examples with SSH

It's a little unsettling to see Packer template files with a clear text password for the vagrant user and root embedded within. Some template authors tell you to delete the vagrant user account if the virtual machine is publicly accessible. Still it's cringe worthy.

In my experiments with Packer I decided to script away some of this cringe worthiness. I took the position that I can improve upon the situation if I
  1. generate my own SSH key pair,
  2. lockout the vagrant user account so that only SSH access is possible using my key, and
  3. encrypt the root password file in the kickstart and preseed files on CentOS, Debian and Fedora.
This isn't perfect but it mitigates the above points as follows:
  1. avoids the use of the vagrant insecure public key.
  2. avoids the use of common words and phrases in example passwords.
  3. limits root password exposure.
As an added benefit, any examples that accidentally make it into production are more secure because the passwords and SSH keys are generated when the Vagrant Boxes are built.

The basic strategy I used to achieve the above is embedded within makefile.credentials. Credentials are generated by default, but can easily be manually created. Credentials are used by the Packer Temple files and a script for generating a Preseed Configuration file.

These examples use Debian but there are Fedora and CentOS examples as well.

Saturday, September 9, 2017

Subprocesses In Python

In A Poor Use of GitPython, I describe how my layering approach in a project using GitPython proved unsatisfactory. Unsatisfactory because I wasn't using GitPython to the full extent of its power. Unsatisfactory because I didn't want to spend time learning Git internals and how GitPython makes them available.

I revisited my approach without GitPython. In A Poor Use of GitPython, my approach resulted in the spread of Git command-line arguments to other functions and I'm looking for a nice abstraction that doesn't cause this problem.

Let's start with subprocess interaction:
1
2
3
4
5
def execute(command, *args):
  """ Use a subprocess to execute a command. Supply the command with any arguments.
  """
  assert 0 < len(command)
  return subprocess.check_output([ command ] + list(args))

I want the output from the command and I want to know whenever I get a non-zero return code. It provides a nice test point for separating my application from the libraries it uses.

I call git using the following function.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
class GitException(Exception):
  """ Throw an exception whenever an error occurs using GIT(1).
  """
  def __init__(self, command, output, returncode):
    assert 0 < len(command)
    self._command = str(command)
    assert 0 <= len(output)
    self._output = str(output)
    assert 0 < returncode
    self._returncode = int(returncode)

  @property
  def command(self):
    return self._command

  @property
  def output(self):
    return self._output

  @property
  def returncode(self):
    return self._returncode

def git(command, *args):
  """ Execute GIT(1). Supply the git command and any arguments.
  """
  assert 0 < len(command)
  try:
    execute("git", command, *args)
  except subprocess.CalledProcessError as e:
    raise GitException(e.cmd, e.output, e.returncode)

Too many layers? Perhaps. All I've achieved thus far is a couple of wrappers that provide strong guarantees on the length of the command. In some respects this is worse than the result I achieved in A Poor Use of GitPython.

The advantage lies in the recognition that some git commands (e.g., git-show-ref and git-ls-files.) return with error 1 under specific circumstances that I might want to handle in higher layers.

Thursday, August 17, 2017

Experiments with Packer and Vagrant

To remedy
Build 'virtualbox-iso' errored: Error uploading VirtualBox version: SCP failed to start. This usually means that SCP is not
    properly installed on the remote system.
Install openssl-clients for Linux guests. Doh.

Friday, August 11, 2017

Blog Entry Syntax Highlighting

I've had good success using Pygments to highlight code on this blog. It provides the best of both worlds: a large selection of language support and it generates standalone HTML.

Standalone HTML was important because many of the other solutions relied upon java script and some of these scripts are hosted on external servers which increased the possibility that the highlighting would disappear whenever the server was down.

I use:
pygmentize -f html -O style=emacs,linenos=1 -o test.html cmd.py
To produce:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
def cmd(path, *args, **kwargs):
    """
    Generate a git log command using the the provided arguments. Empty git log
    lines are stripped by this function.

    Params:
        [in] path - location of repo
        [in] args - non-keyword arguments to provide to the git log command
        [in] kwargs - keyword arguments to provide to the git log command

    Returns: a list of lines returned from the git log command
    """
    repo = Repo(path)
    assert repo.bare == False
    result = list()
    for line in repo.git.log(*args, **kwargs).split('\n'):
        if len(line) == 0:
     continue
        result.append(line)
    return result

The horizontal scrollbar shows up in the codeblock embedded in the file.

 The crude but effective source code to produce the same result:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from BeautifulSoup import BeautifulSoup as beautify
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter
import sys

file = open(sys.argv[1], 'r')

print("""\
<style>
  #codeblock {
    overflow-x: scroll;
    width: auto;
    white-space: nowrap;
  }
""")
print HtmlFormatter().get_style_defs('.highlight')
print("""\
</style>

<div id="codeblock">
""")
formatter = HtmlFormatter(linenos=True)
print highlight(file.read(), PythonLexer(), formatter)
print("""\
</div>
""")
file.close()
Update: GitHub Gists offers a script which greatly simplifies adding code to your blog.

Wednesday, July 19, 2017

Experiments with Vagrant and Packer On Fedora

I've had a strange experience with Packer on Fedora 13 (yes, 13). This may prove interesting to anyone encountered the following error from Packer:
==> virtualbox-iso: Error waiting for SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password keyboard-interactive], no supported methods remain
I'm using Packer v0.10.1.

I found that adjusting ssh_timeout did not seem to have any effect. I ended up setting ssh_handshake_attempts equal to 100.

What I learned in investigating this problem is this is the error message you get when your handshake attempts threshold exceeded. This message implies an authentication problem, but in my case it occurred because SSH wasn't running yet.

To remedy this, confirm whether it's an SSH problem or not by executing:
PACKER_LOG=1 PACKER_LOG_PATH=out.log packer build --debug TEMPLATE
Stop at the step:
==> virtualbox-iso: Pausing after run of step 'StepTypeBootCommand'. Press enter to continue.
Run Packer on a template with headless set to false (the default) so you can watch the virtual machine boot. Login once things are up and running. If the install completes you can also "Press enter to continue". If Packer connects over SSH you can rule out an SSH problem.

To confirm whether it's problem with your Packer template check ssh_wait_timeout and ssh_timeoutssh_wait_timeout is deprecated in favour of ssh_timeout.  Curiously, neither seemed to have any effect on Fedora 13. At one point, I set my timeout to hours and watched Packer shutdown the virtual machine because it couldn't connect is the space of a few minutes.

t was successful in using both parameters on CentOS 7.0 and 7.2 as well as Debian 7.11.0 and 8.5.0. 

The problem doesn't appear to be my Packer template (Fedora uses the same template as CentOS and Debian). 

It isn't Packer (it works properly on CentOS and Debian).

Weird.