|
Written by Juzzy
|
|
Thursday, 28 April 2011 12:35 |
|
As more and more small-time dynamic websites are seeing increased volumes, one begins to ask: How do I scale out and yet keep costs low? While that can seem to be a very simple questions, there's a magnitude of answers. The one I'll address here is database bottlenecks.
Ultimately there is one thing that is making your website / application / whatever slow, that one item is called the bottleneck. You might have 500 things wrong, but usually you only have 1 bottleneck at a time. What happens when you build your web site/application that scales out by web service but still relies on a single database? Well memcache can really help lower the database hits by caching those results from the database and offloading those to a scalable platform. Let's take a look at how I'm doing www.cheap.net:
Simply install memcached, and memcache-php5 on your favorite *nix distro (sorry, no windows memcache)
I personally like php, so I'll be speaking in terms of php, but you can certainly take these concepts and apply them to any other language with memcache client support.
Since I develop on a windows platform, I'll use some logic to determine if I'm in production or dev and my platform:
if ((isset($_SERVER["OS"]) && $_SERVER["OS"] == 'Windows_NT') || ((isset($_ENV["OS"]) && $_ENV["OS"] == 'Windows_NT'))) {
$use_memcache=FALSE;
}
if ($use_memcache)
include("memcache-sessions.php"); //memcache sessions
else
include("session.php"); //standard database sessions
memcache-sessions.php:
<?php
ini_set("session.save_handler", "memcache");
ini_set("session.save_path", "udp://127.0.0.1:11211?persistent=1&weight=1&timeout=1&retry_interval=15");
session_start();
$memcache = new Memcache; //set global object
$memcache->addServer('127.0.0.1', 11211); //add one server, or add multiple ones.
Next I wrap all of my mysql calls into an abstraction layer:
//basic realtime query
function q($q) {
$ret = mysql_query($q)
or die("QUERY: q($q) ERROR: ". mysql_error());
return $ret;
}
//Realtime query that returns all rows into a multi-dim array
function qa($q) {
$items=array();
$ret = mysql_query($q)
or die("QUERY: ($q) ERROR: ". mysql_error());
while($a = mysql_fetch_assoc($ret))
$items[]=$a;
return $items;
}
// So anytime you want to use a select and pull from cache, let's run it through qc() - querycache function qc($query,$time="30"){
global $use_memcache, $memcache;
if (!$use_memcache) {
//echo "DEBUG qc(): $query<br>";
return qa($query);
}
if(stristr($query,"SELECT")) {
//it's a select, let's pull from cache if available...
$key = md5($query);
if ($memcache->get($key)) {
echo "from cache... [$key] $query<br>";
return $memcache->get($key);
} else {
echo "from db...[$key] $query<br>";
// let's grab from db and tuck it into the cache
$result = qa($query);
$memcache->set($key, $result, 0, $time);
return $result;
}
}
else
die("Do not call qc with queries other than selects.");
}
That's the core in a nutshell, in summary, if your website is database driven, but static information, such as a mfg website that's loaded with products and product info, you can crank up the cache expire time to a few days and once it hits cache, you can literally take the database offline and the website will continue to operate. |
|
Last Updated on Monday, 02 May 2011 10:17 |
|
Written by Juzzy
|
|
Friday, 25 March 2011 09:33 |
|
here's a little script if you've ever wanted to timestamp vmstat and let it run overnight or for extended periods of time:
#!/bin/bash
if [ "$1" == "-h" ]; then echo "usage $0 [delay in seconds]"; exit; fi
if [ $# == "1" ]; then
delay=$1
else
delay=1
fi
#print every XX lines
print=30
i=0
while true; do
date=`date +%m/%d\ %H:%M:%S`
if [ $(($i % $print)) -eq 0 ]; then
vmstat | while read line $l; do
echo "$date - $line"
done
else
echo -n "$date -"
vmstat | tail -n 1
fi
i=$(($i + 1))
sleep $delay
done
|
|
Last Updated on Friday, 25 March 2011 09:39 |
|
|
Written by Juzzy
|
|
Wednesday, 16 March 2011 14:15 |
|
Bash bofh - Probably the most common question I get is, why do I need this? And my response is, you don't. In a nutshell all it does is shows you when someone logs into a new shell (that calls bash) and logs their commands to syslog. If you're one of those paranoid, anti-big gov, then sorry this probably offends you. I personally love this patch, not only does it keep up with all 80+ IT users is doing our 300-450+ linux server environment in real time, but it also lets us go back and search old commands that we did a year ago but now have forgotten. I use it for just that exclusively. It's also nice way to figure out what people did to a server if it starts acting funny, or what they did last time when it acted funny.
Now if you're not offended too much, let's dig into how it's done:
I've written a helper script that pulls all the patches along with bash of any version: (required ncftp) Download fetchlatest.sh.
#!/bin/bash
if [ $# -ne 1 ]; then
ncftpls ftp://ftp.gnu.org/pub/gnu/bash/bash-*
echo "Enter the version, ie: 4.2"
read ver
else
ver=$1
fi
mkdir bash-$ver
cd bash-$ver
ncftpget ftp://ftp.gnu.org/pub/gnu/bash/bash-$ver.tar.gz
ncftpget ftp://ftp.gnu.org/pub/gnu/bash/bash-$ver-patches/*
echo "Extracting...."
tar zxf bash-$ver.tar.gz
echo "Patching...."
rm *sig bash-$ver.tar.gz
cd bash-$ver
for x in ../*; do
if [ -f $x ]; then
echo "patching $x"
patch -p0 < $x
fi
done
Now you should be in the actual build directory, let's apply my bash-bofh-4.2.7.patch (currently for 4.2.7)
diff -U 2 -r bash-4.2/bashhist.c bash-4.2_orig/bashhist.c
--- bash-4.2/bashhist.c 2011-03-15 10:00:47.000000000 -0500
+++ bash-4.2_orig/bashhist.c 2011-03-15 09:56:24.000000000 -0500
@@ -32,7 +32,4 @@
#include "bashtypes.h"
#include <stdio.h>
-#include <syslog.h>
-#include <pwd.h>
-#include <sys/types.h>
#include <errno.h>
#include "bashansi.h"
@@ -684,7 +681,4 @@
int force;
{
- struct passwd *pwd_ent;
- pwd_ent = getpwuid(getuid());
- syslog(LOG_INFO, "[%s] %s", pwd_ent->pw_name, line);
if (check_history_control (line) && history_should_ignore (line) == 0)
{
diff -U 2 -r bash-4.2/shell.c bash-4.2_orig/shell.c
--- bash-4.2/shell.c 2011-03-15 10:02:45.000000000 -0500
+++ bash-4.2_orig/shell.c 2011-03-15 09:56:25.000000000 -0500
@@ -40,6 +40,4 @@
#include "filecntl.h"
#include <pwd.h>
-#include <syslog.h>
-#include <sys/types.h>
#if defined (HAVE_UNISTD_H)
@@ -571,10 +569,4 @@
{
char *term, *emacs;
- struct passwd *pwd_ent;
-
- openlog("bash", LOG_PID, LOG_LOCAL5);
- pwd_ent = getpwuid(getuid());
- syslog(LOG_INFO, "interactive shell started by username: %s UID: %d EUID: %d GID: %d EGID: %d",
- pwd_ent->pw_name, pwd_ent->pw_uid, geteuid(), getgid(), getegid());
term = get_string_value ("TERM");
Download bash-bofh-4-2-7.patch.
patch -p0 < bash-bofh-4.2.7.patch
Done! now compile as you normally would, if you can't remember your old ./configure line... Well that will soon be a thing of the past!
Here is the /etc/rsyslog.d/10-command.conf file I use:
local5.* /var/log/commands/commands.log
Make sure you don't let your logrotate your custom logfile!
This should work on bash 4.2, bash 4.2.1, bash 4.2.2, bash 4.2.3, bash 4.2.4, bash 4.2.5, bash 4.2.6, bash 4.2.7 |
|
Last Updated on Thursday, 17 March 2011 15:19 |
|
Written by Juzzy
|
|
Wednesday, 23 December 2009 13:18 |
|
One of the problems with ossec is its a key based security solution that has multiple steps with interactive pieces. I needed to roll this package out to all of our ubuntu servers, so this is what I did:
1. One thing you need to do is remove some of the prompting from the source. Edit the file: ossec-hids-2.2/src/addagent/manage_keys.c
comment out line 91 - 94:
// printf(ADD_CONFIRM);
// fflush(stdout);
// user_input = read_from_user();
and 114-115:
// printf(PRESS_ENTER);
// read_from_user();
2. Compiled and install the package on an ubuntu 8.04 server.
3. I used the script found at: ossec-list to do a quick add using execpt (requires perl modules I could have hacked the source on this one as well, but this was quicker and easier for this article.
4. create ssh keys for your ossec master server (ssh-keygen -t rsa -b 2048)
5. create/append the new key's .pub into your authorized_keys file in the~/.ssh directory
6. go to the server you compiled and installed ossec and prep a directory for packaging:
mkdir /var/tmp/pkg-ossec/
cd /var/tmp/pkg-ossec/
mkdir -p etc/init.d var DEBIAN
cp -r /var/ossec var
cp /etc/ossec-init.conf etc
cp /etc/init.d/ossec etc/init.d/
cat > DEBIAN/control <
Package: ossec
Essential: yes
Priority: required
Section: base
Maintainer: Steve Wakham
Architecture: i386
Version: 2.2
Replaces: ossec (<= 2.2)
Description: Ossec Custom Package
EOF
7. Next we need to create a magic little script to do all of out heavy lifting:
create the file: DEBIAN/postinst (This file gets called once it's done copying the files to /var and/etc)
#!/bin/sh
#if you need to useradd/grpadd ossec, do it here
#we use ldap, so its a known user on our systems
chown -R root:ossec /var/ossec/
chown -R ossec /var/ossec/.ssh
chown -R ossec /var/ossec/queue/ossec /var/ossec/queue/rids
chown -R ossec /var/ossec/logs
chown -R ossec /var/ossec/logs/ossec.log
chown root:root /etc/init.d/ossec
chmod 755 /etc/init.d/ossec
/usr/sbin/update-rc.d ossec defaults
chown root:root /etc/ossec-init.conf
chmod 644 /etc/ossec-init.conf
cat >/tmp/agent_id <
-----BEGIN RSA PRIVATE KEY-----
MIICWwIBAAKBgQDaDaE8g11kVKlLue5vqJ+ETpeaiS+sN7euHrJFjO5OH3l3rHMR
kVKlLue5vqJ+ETpeaiS+sN7euHrJFjO5OH3l3rHMR...
O2Fot16Ht1+5mFon1mtq6uUh7nfEtgAVcA+e8l5jDQ==
-----END RSA PRIVATE KEY-----
EOF
chmod 400 /tmp/agent_id
hostname=`hostname -f`
ip=`/sbin/ifconfig eth0 |grep inet\ addr| sed 's/.*inet addr:\(.*\) Bcast.*/\1/'`
hash=`/usr/bin/ssh -o StrictHostKeyChecking=no ossecmaster.domain.com -i /tmp/agent_id "/root/ossec-mkclient.pl $hostname $ip"`
rm /tmp/agent_id
/var/ossec/bin/manage_agents -i $hash
/var/ossec/bin/ossec-control start
8. That's it, dpkg -b pkg-ossec ossec-2.2-i386.deb and to install dpkg -i ossec-2.2-i386.deb
I know this can be a security problem by leaving keys inside of a package, there's a number of measures you can take to reduce risk in this area, we created a special user and when you login you're dropped into a chrooted enviroment with only manage keys binary that's bound to the real one and suid'd to write to files the user cannot read, but worse case, yank your key once you deploy and just manually enter it from then on. |
|
Last Updated on Tuesday, 04 May 2010 08:19 |
|
