October 15, 2012

Fixing Helpspot's incorrect gziping of css and javascript

Ran into a strange gzip encoding bug with HelpSpot my preferred Help Desk solution where it would return gzip encoded content for it's css and javascript php combiner scripts even when Nginx was configured to not gzip content, compressing outbound data was disabled from php (confirmed via phpinfo()). Confirmed that using curl returned the uncompressed data.

Passing a HTTP header HTTP_ACCEPT_ENCODING set to none back to HelpSpot via nginx's fastcgi parameter solved the issue of HelpSpot deciding to gzip data for it's css and javascript file combiner php code:

Turning gzip compression on inside nginx works as expected now with HelpSpot's css and javascript combiner scripts.

September 1, 2012

Reverse Proxying Confluence and JIRA

At my $dayjob we use various products from Atlassian. Following the latest cohort of the JIRA security advisory, we needed a way to lock down access to our JIRA via our reverse proxies.

Seeing we use Apache I placed the usual Apache directives in the configuration file for users to authenticate against the company replicated LDAP pair I had setup previously. Later it was reported some folks were having issues with a tomcat error page relating to basic authentication from when we used an outsourced IT company to handle user management they decided to drop the email prefix so the added awesome side effect of users being logged into JIRA or Confluence when getting past the HTTP Authentication dialog box required another Apache directive to not forward the Authorisation bits onto Tomcat.

The directive is:

March 19, 2011

Learning from others mistakes - move configs out of your boot loader

I'm a keen believer of learning from other folks mistakes and attempting to not repeat them. Source code disclosure and even worse configuration disclosure which happened with tumblr. One thing to remember, if you keep your configuration for your app outside your webroot, you reduce the chances of you actually accidentally disclosing your configuration. Typically one could even do something like:

require_once dirname(dirname(__FILE__)) . '/bootstrap.php';

Rather than having everything in your index.php file. Separating your configuration directives out to a separate configuration file outside the webroot and modifying that when making changes.

One can use your version control pre commit scripts to check that you've not borked a the file - checking for php scripts being of a file type php script vs file is one way of doing this - foo.php contains a php script starting with "i?php" and bar.php has same code starting with "

$ file foo.php 
foo.php: PHP script text
$ file bar.php 
bar.php: ASCII text

January 3, 2011

Atlassian JIRA and URL rewriting

Prior to JIRA 4.2.2, there was a bug JSP-70163 which can be easily fixed by telling apache to rewrite the URL when you use a different base via tomcat i.e. /jira instead of / when doing https://hub.yourdomain.com/jira/

To fix the issue where screenshots get uploaded and don't redirect properly you can make use of apache's redirects to send anything destined prefixed by /browse to /jira/browse by adding the following to your virtualhost config:

Redirect /browse https://hub.yourdomainname.com/jira/browse

November 2, 2010

Locking down rpcbind on OpenSolaris

Every now and again one is asked how one can lock down rpcbind from with a shared-ip zone without being able to use ipfilter from within the zone. Fortunately tcpwrappers save the day by allowing us to enable rcpbind to use the tcpwrapper functionality to lockdown who can connect to port 111 (the port which Sun RPC use).

You have run rpcinfo against a zone and it returns numerous lines saying that rpcbind and additional rpc based serivces are available remotely.

rpcinfo -p IP.AD.DR.ESS
   program vers proto   port  service
    100000    4   tcp    111  rpcbind
    100000    3   tcp    111  rpcbind
    100000    2   tcp    111  rpcbind
    100000    4   udp    111  rpcbind
    100000    3   udp    111  rpcbind
    100000    2   udp    111  rpcbind
    100024    1   udp  62025  status
    100024    1   tcp  36015  status
    100133    1   udp  62025
    100133    1   tcp  36015
    100021    1   udp   4045  nlockmgr
    100021    2   udp   4045  nlockmgr
    100021    3   udp   4045  nlockmgr
    100021    4   udp   4045  nlockmgr
    100021    1   tcp   4045  nlockmgr
    100021    2   tcp   4045  nlockmgr
    100021    3   tcp   4045  nlockmgr
    100021    4   tcp   4045  nlockmgr

For tcpwrappers to allow and deny traffic to rpcbind, we need to create two files namely hosts.deny and hosts.allow both under the /etc directory.

echo "rpcbind: ALL" >> /etc/hosts.deny
echo "rcpbind: 10. 192.168. 127." >> /etc/hosts.allow

Once this has been done, we are now ready to enable tcpwrappers to disallow access to rpcbind:

svccfg -s rpc/bind setprop config/enable_tcpwrappers=true
svcadm refresh rpc/bind

One can also run a quick script which I quickly put together that will run the four commands mentioned above for you:

curl http://github.com/jacques/adminscripts/raw/master/lockdownrpcbind.sh | sh -x

One can verify that rcp/bind is sitting behind tcpwrappers using by running:

# svcprop -p config/enable_tcpwrappers rpc/bind

And remotely one can verify this by running:

# rpcinfo -p IP.AD.DR.ESS
rpcinfo: can't contact portmapper: RPC: Rpcbind failure - RPC: Failed (unspecified error)

October 30, 2010

Getting java to trust a SSL Certificate

After fixing a few bugs in vimfluence, I tried to connect edit some wiki pages on my personal Confluence wiki which sits behind a self-signed SSL certificate. It works something like:

$ personalwiki HOME
Unable to log in to server: https://hub.example.com//rpc/soap-axis/confluenceservice-v1 with user: username.
 Cause: ; nested exception is: 
	javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
could not close pipe at ./vimfluence line 94.

A bit of an annoyance with Java is that you have to tell Java that it has to implicitly trust a self-signed SSL certificate if you want to connect to it via some jar. In my case I'm using vimfluence to connect to a wiki sitting behind a self-signed SSL certificate. To import the certificate into the keystore, one uses a binary called keytool which brings back strange memories of Actaris's biz.switch documentation from a previous joblet.

(cd /Library/Java/Home/lib/security && sudo keytool -import -keystore cacerts -alias example-wild -file ~admin/example-wild.crt)

One can easily get the SSL certificate by running:

openssl s_client -connect hub.example.com:443

Some gsed magic to insert a line below the regex matched text

So I'm working on finishing automating as much of my install fresh installations of Crowd, Confluence and JIRA. One of the things I was looking for is how to use gsed to insert some thing after a specific line has been matched with regex. For example if I insert a line before something I used something.

Before I started testing I created a test text file:

[jacques@auriga ~]$ cat far

Then I wanted to insert the word testing above the word ABOVE:

[jacques@auriga ~]$ gsed -i"" -e "/ABOVE/i\
testing" far
[jacques@auriga ~]$ cat far

So that works well - after quite a bit of fiddling it was discovered to insert a line after something I needed to change the i script modifier to an a:

[jacques@auriga ~]$ gsed -i"" -e "/BELOW/a\
testing" far
[jacques@auriga ~]$ cat far

Getting Crowd 2.0.7 to behave itself under SMF

I was chatting to Jerry Jelinek during the week on IM about an issue that I was working on debugging - the reason Crowd 2.0.7 was not behaving under SMF - it was being randomly killed off and never transitioning to an online state even though it's start_crowd.sh script had cleanly exited with an exit code of zero.

Looking at the scripts that Confluence and JIRA comes both of those start tomcat up with passing the argument of start to the catalina.sh script which starts tomcat up.

When doing a "svcs -p crowd" it would keep showing that the state was "offline*" - randomly it would get killed and I would have a cronjob running every minute to clear the service while digging into this.

Turns out the dear Atlassian folks regressed and put in start_crowd.sh a "run" instead of "start" for apache tomcats catalina.sh script which due to the way it starts java SMF thinks that it is still running the command and will zap the java process when the timeout comes around. So to fix this, one has to edit the start_crowd.sh and change run to start and then one is good to go. Crowd starts behaving itself while under SMF and life is good once again.

The manifests which I use for Crowd, Confluence and JIRA are available on GitHub.

October 28, 2010

Debugging an Apache 403 error and .htaccess issues

Seems that apache can have some misleading logfile entries. When you see something about:

"cfg_openfile: unable to check htaccess file, ensure it is readable"

You can either be out of file handles or the parent directory is not readable or infact the .htaccess file is not readable by the user running apache.

October 16, 2010

Preventing source code in your .git directory being viewed

Adding some lines to your apache configuration file can prevent people from viewing files under your .git directory or files like .gitmodules. Depending on whether you have other files/directories which you to stop folks reading, you can easily add more similar directives.

I'm running a git clone of flamework for testing some stuff and was looking to see if the I could view files under the .git directory. So I looked to see what files exist under .git and chose HEAD and went to my browser and pulled up http://flamework.example.com/.git/HEAD and it showed the file contents. After adding the snippet above to my apache config, I was no longer able to view files under the .git directory or the .gitmodules files.

September 5, 2010

Crowd randomly dying issues

Dear Lazyweb,

Does anyone have a solution for Atlassian Crowd keeping on crashing. I recently upgraded from 2.0.6 to 2.0.7 and that is were the issue started. Crowd randomly crashed every couple of minutes after startup. I have a cronjob that runs svcadm clear crowd every minute to try and keep the downtime to a minimum. Is there something like xdebug (for php) for java?

May 18, 2010

OpenSolaris and VMWare boot issues

When running OpenSolaris SX:CE >= snv_100 on different flavours of VMWare (Fusion, ESX, etc.) one needs to change edit the grub menu to include "disable-pcieb=true" at the end of the kernel line. To do this you will need to hit "e" to edit on the grub menu item, and then "e" on the line for the kernel. At the end of the line if there is no "-B" with options after that, add the "-B disable-pcieb=true". Once you have installed OpenSolaris, you can go ahead and edit your /boot/grub/menu.lst and add this there as well.

December 22, 2009

How many drafts do you have in your blog software?

I currently have 43 draft entries in my blog software (movable type) that I've not yet published as I have not gotten round to posting some of the posts.

December 3, 2009

Google "Public DNS" to compete against OpenDNS

Google have launched their "Public DNS" service - a similar service to the OpenDNS public dns resolvers. It is going to be interesting to see how much competition occurs. It is usually faster to use the OpenDNS public dns resolvers than one's upstream ISP's one. Not sure when Google will start adding additional features like OpenDNS like blocking access to malware sites, ability to see how many DNS queries you are making, etc.

If you want a non-filtered service at this point in time use the Google Public DNS servers (they are using Level3 for their upstream connectivity): and

And if you like filtered DNS results you would want to use the OpenDNS ones: and


Quick stat from doing DNS queries against Google, IS and OpenDNS's cache servers:

Google: 546ms and 243ms
IS: 303ms and 36ms
OpenDNS: 432ms and 297ms

Looks like all three cache the results on their anycast clusters.

How do you use MySQL as a DBA?

Interesting question for MySQL DBA's: do you prefer using the mysql cli interface and know the SQL statements to execute in order to do your admin work like creating databases or do you prefer using the mysqladmin interface? Personally I prefer using the mysql cli interface and executing things like 'CREATE DATABASE foo' rather than 'mysqladmin create blah' - I suppose I like the SQL queries over remembering the other way of doing things.

November 9, 2009

Upgraded Movable Type

Finally managed to get round to upgrading movable type - need to hack my wpcp function to add files that haven't been added with svn add for some odd reason. I must admit that I like the cleaner appearance of the admin interface.

Added bonus is not having to spend a few hours upgrading the templates - it seems that they've fixed that broken feature this time around.

August 11, 2009

jot gotcha

Seeing that I run into this jot gotcha every couple of months while writing shell scripts, I thought I would add a reminder for myself here (and so that google can pick this up for me next time).

On FreeBSD, by default you don't get access to the seq command to generate sequences, so you naturally will use jot for your shell scripting needs for generating number sequences.

So for example, we want to start with 0 and increment to 4 (one cannot use variables in the bash sequence generator, so {0..$max} won't work here. So for example we have 5 logfiles that we want to grok on a server and another has 6 so instead of hardcoding the varible on each server like {0..4} where we have 4, {0..6} where we have 6, we use jot and some nasty piping commands to get that number.

root# jot 4     

Okay so misread the manual very quickly and try:

root# jot 0 4

Pass the number of iterations (5), start (0) and where it ends (4):

root# jot 5 0 4

July 4, 2009

Firefox 3.5 Regression issues with wildcard SSL certificates

I really hate educating people how to do the following, but a bit of an explanation around this. Mozilla Firefox 3.5 now does not play nice with wildcard SSL certificates where you do things like service.servername.example.com when you only have a wildcard SSL certificate for *.example.com - the guys at Mozilla have marked the bug as "Won't Fix" which basically means that one now needs to teach people how to bypass this warning.


The latest version of Mozilla Firefox 3.5 has issues with wildcard SSL certificates and is giving warnings that "This Connection is Untrusted" when visiting https://service.servername.example.com/ - this is unfortunately an issue with the Mozilla Firefox 3.5 web browser. You will need to click on the "Add Exception..." link, ignore the warning about "You are about to override how Firefox identifies this site. Legitimate banks, stores, and other public sites will not ask you to do this.". Click on "Get Certificate" and then click on "Confirm Security Exception". A bug is being filled with Mozilla about this regression in their software.


Hopefully Mozilla can release a patch Firefox 3.5.1 which reverts this regression in their software.

Anyway here is the list of bugs I'm going to be keeping an eye on for this issue:

July 3, 2009

Deploying sites in seconds

Currently playing around with the private beta of the Smart Platform from Joyent - took less than 2 minutes to deploy their hello world app (apart from me getting a bit annoyed with git). Need to still work on getting git to not warn me about:

$ git push
warning: You did not specify any refspecs to push, and the current remote
warning: has not configured any push refspecs. The default action in this
warning: case is to push all matching refspecs, that is, all branches
warning: that exist both locally and remotely will be updated.  This may
warning: not necessarily be what you want to happen.
warning: You can specify what action you want to take in this case, and
warning: avoid seeing this message again, by configuring 'push.default' to:
warning:   'nothing'  : Do not push anything
warning:   'matching' : Push all matching branches (default)
warning:   'tracking' : Push the current branch to whatever it is tracking
warning:   'current'  : Push the current branch
Everything up-to-date

A basically static site took a while longer as I misread a page in the documentation and then had files sitting in the incorrect location. One still needs a bootstrap.js file even if you intend on just serving up static files.

Seems like one needs to tell git to only attempt to push the currrent branch using the following command gets rid of the warning when using git push in future:

$ git config push.default current

April 18, 2009

FreeBSD vr network card issues

Sort of a note to self for the next time I bump into a very annoying issue with network cards that use the VIA Technologies Rhine I/II/III controller chips on network cards like the quite popular dlink (shows up under vrX in ifconfig -a) is that when you start binding multiple IP addresses on a vr card a few seconds later the card stops working.

Turns out there is a bug of sorts with the FreeBSD vr driver which needs some fixing so moral of the story for the moment is not using vr based network cards till this is fixed.

March 4, 2009

Twitter Search the posterchild for 500's

One of my pet peeves is currently when Twitter Search is down - there goes being able to zone into conversations on any given topic from bsdisms, following cloud computing, etc.

Tired of seeing:

Status: 500 Internal Server Error Content-Type: text/html 500 Internal Server Error

How difficult is it to fix something that was not broken when it was Summize?

This is what a reply from twitter looks like:

HTTP/1.1 200 OK
Date: Wed, 04 Mar 2009 20:52:44 GMT
Server: hi
Content-Type: text/html; charset=UTF-8
Cache-Control: max-age=300
Expires: Wed, 04 Mar 2009 20:57:44 GMT
Vary: Accept-Encoding
X-Varnish: 1089043412
Age: 0
X-Cache-Svr: searchweb008.twitter.com
X-Cache: MISS
Via: 1.1 varnish, 1.1 bc1-rba
Content-Length: 122
Connection: Keep-Alive
Set-Cookie: _search_twitter_sess=*snipped*; path=/

Status: 500 Internal Server Error
Content-Type: text/html

500 Internal Server Error


Okay it's up if I go via reverse proxy in the USA - so it's just down for South Africans.


IS have added some rules to their netcaches and search.twitter.com is working again.

February 22, 2009

No Smarty

If you've not heard about it yet, there is now a website advocating not using Smarty. The time for using smarty has gone - but makes one wonder if Flickr, Facebook, etc. are still using it.

January 1, 2009

Happy New Year!

Just a quick blog post to say Happy New Year! 2008 seems to have flow by quite quickly and at the start of 2009 I've made a resolution to not make any resolutions as I never tend to be able to keep any of my new years resolutions. Hopefully this year I can get round to blogging more often which is something which I've not been following through on.

October 26, 2008

MRTG missing SNMP_util

One of the joys of installing ports on FreeBSD is the odd occasion that the dependencies list is missing a dependency. is not installed you get nice perl errors in the case of the following

# cfgmaker public@
Can't locate SNMP_util.pm in @INC (@INC contains: /usr/local/bin/../lib/mrtg2 /usr/local/bin /usr/local/lib/perl5/5.8.8/BSDPAN /usr/local/lib/perl5/site_perl/5.8.8/mach /usr/local/lib/perl5/site_perl/5.8.8 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.8/mach /usr/local/lib/perl5/5.8.8 .) at /usr/local/bin/cfgmaker line 105.

Turned out SNMP_util is missing and required a quick install of SNMP_Session to make cfgmaker work.

cd /usr/ports/*/p5-SNMP_Session
make install clean

One can see that the dependency list for mrtg excludes SNMP_Session:

# pkg_info -rR mrtg-2.16.2,1
Information for mrtg-2.16.2,1:

Depends on:
Dependency: perl-5.8.8_1
Dependency: p5-Socket6-0.22
Dependency: p5-Pod-Parser-1.35_2
Dependency: p5-IO-Socket-INET6-2.56
Dependency: p5-Digest-SHA1-2.11
Dependency: p5-Digest-HMAC-1.01
Dependency: p5-Crypt-CBC-2.30
Dependency: p5-Crypt-DES-2.05
Dependency: p5-Net-SNMP-5.2.0
Dependency: png-1.2.32
Dependency: jpeg-6b_7
Dependency: pkg-config-0.23_1
Dependency: freetype2-2.3.7
Dependency: libiconv-1.11_1
Dependency: gd-2.0.35,1

Anyways I need to send-pr a patch for the port to get this fixed in the ports tree.

September 8, 2008

Patches to update rails 2.1.0 to 2.1.1

I've made some patches to update Ruby On Rails version 2.1.0 to 2.1.1 where you make use of the unpacked gems and a subversion repository (running 'rake rails:freeze:gems' has a bad habit of leaving your working copy in a mess as it blows away your 'vendor/rails' directory and recreates it).

Download the patch files by using the following script and apply them into each directory under vendor/rails by running rails-updater.sh.

If you have issues with patching files (like when using svn eol-style:native for files) you may need to grab the failed files by doing 'gem unpack component' and copying those files over and removing the reject files. I've tested the patches on a couple of open source apps and it applied cleanly except to one working copy.

patching file CHANGELOG
patching file Rakefile
patching file lib/action_mailer/base.rb
patching file lib/action_mailer/vendor/text-format-0.6.3/text/format.rb
Hunk #1 FAILED at 1.
1 out of 1 hunk FAILED -- saving rejects to file lib/action_mailer/vendor/text-format-0.6.3/text/format.rb.rej
patching file lib/action_mailer/vendor/tmail-1.2.3/tmail/parser.rb
Hunk #1 FAILED at 12.
1 out of 1 hunk FAILED -- saving rejects to file lib/action_mailer/vendor/tmail-1.2.3/tmail/parser.rb.rej
patching file lib/action_mailer/version.rb
patching file test/abstract_unit.rb

Another way to fetch the files is to fetch them from the github website using url's like:


i.e. for the failed lib/action_mailer/vendor/text-format-0.6.3/text/format.rb mentioned above we would fetch it from the following URL - http://github.com/rails/rails/tree/v2.1.1/actionmailer/lib/action_mailer/vendor/text-format-0.6.3/text/format.rb?raw=true.

September 3, 2008

Exim gotcha with SMTP Auth

One gotcha when using exim to do authenticated SMTP one wonders why you keep seeing the following:

Return-path: <"email@add.re.ss"@server.host.name>
Sender: "email@add.re.ss"@server.host.name

One needs to modify your acl for acl_check_rcpt for authenticated SMTP connections to contain the sender_retain bit like below:

  accept  authenticated = *
          control       = submission/sender_retain

Restart your exim (in my case on FreeBSD I used /usr/local/etc/rc.d/exim restart) and send a email:

Return-path: <email@add.re.ss>
Sender: email@add.re.ss

August 29, 2008

SA Pro Podcast from Ben Rockwood

One of my work colleagues, the ubergeek Ben Rockwood has started a podcast called SA Pro a podcast for Systems Administrators. In the first episode (episode zero) Ben chats together with with Joe Moore of Siemens and Mark Imbriaco of 37signals.

In the podcast we'll use one of two formats, classic 1-on-1 interview style and a round-table discussion format. This episode is the latter.

Together with Joe Moore of Siemens and Mark Imbriaco of 37signals we discuss the following questions:

  1. What is the mark of a good SA?
  2. What are the essential qualifications?
  3. Does formal education and/or certs matter?

Whats really new and unique is that Joe, Mark, and I don't know each other. They both responded to a request for participants on the OpenSolaris SA's list and matched the qualifications I was aiming for, thats the extent of it. This is interesting because even though the three of us are in very different circumstances, have different histories, and are geographically separated, we're not very dissimilar. It amazes me how much unity there is among a group with so few governing institutions.

August 13, 2008

37 Signals Live : Episode 2

Jason Fried and David Heinemeier Hansson in the kitchen at 37 Signals offices.

Jason Fried and David Heinemeier Hansson broadcast to 883 people from out of the kitchen at the 37 Signals offices. Audio is coming shortly. Please note that the audio is a bit choppy (due to broadcasting issues and bandwidth issues with Verizon in South Africa). I've normalised the audio as well as tried to remove as much noise as possible.

Feel free to download a copy of the audio file of the 37 Signals Live : Episode 2. The audio is being served from BingoDisk.

August 9, 2008

So long and thank you for all the fish, PHP4

It's finally EOL time for PHP4 and they sneaked out a release of PHP 4.4.9 out a day later. Not too many changes to 4.4.9.

August 5, 2008

37Signals Q&A

An interesting way of engaging with their community by doing a live Q&A session with their users online (with users submitting questions via an IRC interface):


I've removed the video flash player - it seems that the 37 Signals guys did not record the Q&A session. I've found the live replay over here.

About Me

Jacques Marneweck

Oneliner: Jacques Marneweck is a systems administrator / DBA with a background of administering and developing on a FreeBSD/OpenSolaris platform using PHP, perl, ruby, MySQL, nginx and apache.


October 2012
Sun Mon Tue Wed Thu Fri Sat
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31      


Creative Commons License
This weblog is licensed under a Creative Commons License.
Powered by
Movable Type 4.38.

Hosted at