Philip: music, code and stuff.

Notes on a 1987 Sampler's Firmware

Sun, 31 May 2026 00:00:00 +0000

Notes on a 1987 Sampler's Firmware

2026-05-31 casio music ai

The Casio FZ-1's firmware is 64 KiB of V50 machine code, masked into two 32 KiB ROMs. It runs the keyboard, the LCD, the floppy drive, and an eight-voice sound engine scheduled on a single timer interrupt. I have been reverse engineering it. I had never tried it before this. It turns out I really enjoy it.

Reverse engineering involves taking compiled bytes back to something you can read and reason about. There is no source code, no comments, no symbol table; just instructions, addresses, and the side effects they cause.

I downloaded Ghidra, the disassembler the NSA open-sourced in 2019. You open a binary and it gives you an instruction listing, a decompiled C view that approximates the original, and a graph of which functions call which.

The first thing the disassembler told me was nonsense. The reason was that the FZ-1's ROM lives on two physical chips wired so one provides the low byte and the other the high byte of every 16-bit word the CPU fetches; neither half makes sense on its own. The two have to be zipped together first. These are the last 16 bytes of the merged image:

fz1s.bin (even bytes):  EA 06 F0 49 4F 55 49 ...
fz2s.bin  (odd bytes):  50 00 48 52 59 4B FF ...
interleaved:            EA 50 06 00 F0 48 49 52 4F 59 55 4B 49 ...

The first five bytes shown, EA 50 06 00 F0, decode as JMP FAR F000:0650: the reset vector. They sit at the very top of the address space, which is where 8086-family CPUs boot from. The next eight, 48 49 52 4F 59 55 4B 49, spell HIROYUKI. Apparently the firmware engineer's name. Nobody seems to have confirmed publicly who he was; the same signature was spotted on a Gearspace thread years ago and the trail goes cold there too. Someone else figured the interleave out on that thread; I rediscovered it independently and we both ended up staring at the same name.

The first stretch of work was hand-rolled. I built a small pipeline that fed the model pre-decoded instruction windows for one function at a time, with whatever annotation I already had for context. The model proposed an explanation. A separate gate then re-read the same bytes out of the ROM image and rejected any claim whose cited bytes did not match what was actually there. I ran this in batches, eight functions at a time, until every callable routine in the firmware had at least a one-line summary tied to specific instructions.

Then I switched to ghidra-mcp, which exposes Ghidra's API to a model directly. Instead of feeding it pre-decoded windows, the model could ask Ghidra to decompile a function on demand, list its callers, follow a cross-reference, write its annotations back into the Ghidra project rather than into a sidecar file. The C the model produced was closer to what the original probably looked like, and it was easier to spot where the model was confabulating versus where it was reading the bytes, because the bytes were one query away. Or more honestly: more than once the model wrote a plausible-sounding paragraph that the gate then rejected because the citations did not survive contact with the ROM.

What I keep coming back to about the firmware itself is how often one routine does the work of several. There is a routine called work_reset that the firmware uses for the cold boot and for every soft mode switch; one piece of code zeroes the sound engine and fills the voice table with free markers, called from seven different sites. Every LCD drawing primitive (pset, line, cls) takes a colour argument whose top bit, if set, suppresses the LCD write and composes into a 1-bit-per-pixel mirror in RAM instead. So pset(x, y, color) flushes to the display, and pset(x, y, color | 0x8000) accumulates into the mirror, which is then committed in one burst by a later call without the bit set. ORing one bit into a parameter turns immediate drawing into batched drawing. The voice allocator, when a MIDI note arrives, walks the eight voice slots up to three times: first for a free one, then for a slot already playing the same key on the same channel (replacement), then for the oldest live voice (stealing). And the data formats themselves are unified: LOAD and SAVE each go through one orchestrator that dispatches to three transports (floppy, the inter-unit data port, MIDI SysEx), with the same buffer composition and chunking. A voice saved to floppy and the same voice broadcast as MIDI SysEx are the same bytes in the same order. One engineer worked this out forty years ago on a machine with 64 KiB of code budget, and the taste is consistent from the cold boot to the LCD primitives to the voice scheduler to the way bytes leave the box.

The practical reason for going to all this trouble: I needed a sharper model of what the firmware actually expects from a disk image before I could trust fizzle's output beyond the cases I had already tested by hand, and I wanted to write small Type-5 programs (Casio's term for user-loadable binaries that run on the sampler) without each one being a fresh trial. Both of those are closer now. The disk-format work is paying off because the firmware-side reader is no longer a black box. More to come soon.

Thank you, HIROYUKI.

Tools for a 1987 Sampler

Fri, 29 May 2026 00:00:00 +0000

Tools for a 1987 Sampler

2026-05-29 casio music ai

Recently I wrote about the gap between my Casio FZ series sampler and the tools for getting samples onto it. fizzle is what I have been making to fill it. It turns modern sample files into disk images a Casio FZ can read.

A tool for this kind of job already existed, just not for the machine I cared about. Chicken Systems Translator converts between sampler formats, and I use it to move SFZ instruments onto my Akai S3000XL. The Casio FZ series is not on its list. Awave Studio reads and writes individual FZ files but not the disk format, and only on Windows. So the samples that played on the Akai would not play on the Casio, even though the Casio is the more interesting instrument to my ears.

There was another reason to do it myself. I wanted to experiment with pair programming with an AI, in this case Claude Code, and a binary format suits that well: either the disk loads on the sampler or it does not, so I always knew whether a change had worked.

Go was a good fit. Cross-compiling a static binary for macOS, Linux and Windows is one command each, and most of the libraries I used are MIT licensed, so I can hand the binaries to someone else without much thought.

I ran Casio's 1987 FZ-1 Data Structures document through OCR, fed it to the model, and we wrote code against it. Then I put a disk image onto the Gotek floppy emulator in my FZ-10M and saw what the sampler made of it. Some of it matched the document. Some of it did not, and the front panel of an FZ-10M is small, with one corner of mine hard to read, so working out why was slow. Then we fixed something and went round again.

For most jobs the file that matters is the fzf. An fzf (full dump) holds up to 64 samples with a bank mapping over the top that decides which key plays which sample, on which channel, through which output. The Casio loads one of them in a single operation, which is easier than loading voices one at a time.

A big instrument does not always fit on one floppy, though, and splitting it across two was the part that gave me trouble. The FZ only has 2 MB of sample memory, so a large patch has to span two disks. My first attempt put a bank sector and voice headers on both disks, on the assumption that each disk ought to describe itself. The sampler loaded both disks, but the audio from the second disk came in with a glitch at the start that sounded like clipping. At first I assumed one of my own samples was clipping.

Claude Code and I went round this for a while. Every idea it had was reasonable, and none of them were right, because the answer was not in the spec or in my code. I only worked it out by saving a two-disk set from the FZ-10M itself and reading the bytes. The second disk had no bank sector and no headers at all. It was just the leftover audio from the first disk, and disk one carried the parameters for the whole instrument, including the voices whose audio ended up on disk two. The headers I had been writing to the front of my second disk were what the sampler was playing as that glitch. Once I understood that, the fix was small. The model could work from the format I had described to it, but it could not know the one thing I had never told it, because that thing only existed in what the hardware wrote to the disk. I had to go and find it.

fizzle also goes the other way now: an SFZ instrument becomes an fzf, and an fzf can be taken back apart into SFZ with its WAVs, so a patch made on the hardware can come back into a DAW. Round-tripping turned out to be a good way to catch mistakes. If a file does not come back the same as it went in, something is wrong, and you notice straight away.

Once it was handling real instruments, I gathered a pile of old FZ disks from around the internet and ran the lot through fizzle at once. That found problems no single disk had shown me. The 1987 document is mostly right, but it is not the same as the disks people actually have. One field I had treated as a voice count was really a count of key splits. Most of the time those are the same number, so it never mattered; on the disks where they differ, my code fell over. Others had small corruptions that probably crept in years ago, when someone copied a floppy onto a hard drive.

Working this way was good, though a few things grated. Sometimes it ignored instructions I had written down plainly: I would set a rule and watch it get broken a few messages later. More than once it tried to get a failing test to pass by quietly weakening the test instead of fixing the code, which is the last thing you want from something meant to be keeping you honest. So I learned to keep the strict checksum tests in place and to read the diffs myself rather than take its word for what it had done.

It still sped the work up. A fiddly byte layout never bored it, and it was good company for the slow parts late at night. It did not save me from having to understand the problem, though. The two-disk bug is the clearest example: the answer was not written down anywhere, so it had to come from me.

The samples that play on my Akai play on the Casio now too, and the FZ-10M has gone from a fussy curiosity to something I can actually load with my own sounds. There are more tools on the way, and the short video I promised last time is coming soon.

Notes on a 1987 Sampler

Thu, 14 May 2026 00:00:00 +0000

Notes on a 1987 Sampler

2026-05-14 music casio

The Casio FZ-1, released in 1987, is a 16-bit sampler with eight voices, three sample rates, and a 3.5" floppy drive. The FZ-10M rack came with 2MB of sample memory as standard, double what the FZ-1 keyboard shipped with.

You can hear an FZ-10M on LFO's Frequencies (Warp, 1991). Mark Bell and Gez Varley used it on the title track. Aphex Twin used one on a lot of his early records. The FZ series came out of Casio's late-eighties professional line, alongside the calculators and the VL-Tone.

The samples are mono, the envelopes take a while to learn, and the filters smear the signal in a way that is hard to describe and very pleasant to hear.

uber weird filter architecture, dig/analog hybrid

Aphex Twin, SYROBONKERS 2014

The thing about owning an FZ in 2026 is that getting a sample onto one is its own small project. You can put a real 3.5" floppy through a real drive, assuming the drive still works and you can find one of the disks the firmware will read. More often you fit a Gotek floppy emulator and load .img files from a USB stick. Either way you need the disk image. The disk format is from 1987 and has not had a lot of attention in 2026's tooling landscape. There is a primary document somewhere in a forum thread, there is a sampler in my rack, and not much tooling that bridges the two.

Some tooling does exist for the file format. Awave Studio, a long-running Windows application for converting between sampler formats, reads and writes individual FZ files: 16-bit mono samples, loop points, names, instruments, and start and end offsets. Envelopes and LFOs are not supported, and neither is the FZ disk format directly. You can move samples in and out at the audio level, but the parts that make an FZ sound like an FZ are still on the other side of the gap.

I have been spending time in that gap. Jacob Vosmaer wrote about the same problem a few years back, and some of his work was a starting point for mine. The loop is simple: read the spec, write the code, load the disk image onto the Gotek, and check what the bytes look like on the sampler's front panel. Repeat. It's a bit tedious. The FZ-10M's front panel is small to begin with, and one corner of mine is hard to read. Some of it lines up exactly with the 1987 document. Some of it does not. What I've been working on should be ready in a few weeks. There will be a short video to go with it.

Testing distributed-process Apps Using Hspec

Mon, 24 Dec 2018 00:00:00 +0000

Testing distributed-process Apps Using Hspec

2018-12-24 distributed-process haskell hspec testing

distributed-process is a Haskell library that brings Erlang-style concurrency to Haskell. Whilst developing an application at work that uses it, I found that there wasn't much material online describing how to test distributed-process applications. I used some techniques from object-oriented programming that allowed me to test the behaviour of my application whilst I was learning how it was supposed to fit together. This post documents some techniques I found useful.

Application

Our example revolves around a fairly simple client-server application. The client process can send data to the server and output responses to the console, whilst the server performs calculations and sends the results back to clients.

{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE NoImplicitPrelude #-}
{-# LANGUAGE TupleSections #-}


module Main where


import Control.Distributed.Process (Process, ProcessId, expect, getSelfPid, register, say, send, whereis)
import Control.Distributed.Process.Node (LocalNode, initRemoteTable, forkProcess, newLocalNode, runProcess)
import Data.Binary (Binary)
import Network.Transport (Transport(..))
import Network.Transport.TCP (createTransport, defaultTCPParameters)
import Prelude (String)
import Protolude
import Test.Hspec


-- APP


app :: LocalNode -> Process ()
app node = do
  _ <- newProcess node "client" clientProcess
  _ <- newProcess node "server" serverProcess
  pure ()


-- PROCESSES


data ClientMsg =
      Ask [Int]
    | Result Int
    deriving (Eq, Generic, Show)

instance Binary ClientMsg


clientProcess :: Process ()
clientProcess = forever $ do
  msg <- expect

  case msg of
    Ask ints -> do
      self <- getSelfPid
      namedSend "server" $ Calc self ints

    Result n ->
      say $ "received: " <> show n


data ServerMsg =
  Calc ProcessId [Int]
  deriving (Eq, Generic, Show)


instance Binary ServerMsg


serverProcess :: Process ()
serverProcess = forever $ do
  msg <- expect

  case msg of
    Calc sender ints -> do
      send sender $ Result (sum ints)

The code above uses some helper functions that aren't present in distributed-process; these are included in the code below to let you follow along at home.

-- PROCESS HELPERS


-- | Fork process and register it
newProcess :: LocalNode -> String -> Process () -> Process ProcessId
newProcess node name process = do
  pid <- liftIO $ forkProcess node process
  _ <- register name pid
  pure pid


-- | Create a new transport
newTransport :: IO (Either IOException Transport)
newTransport =
  let
    host =
      "localhost"
    port =
      "3000"
  in
    createTransport host port (host,) defaultTCPParameters


-- | Spins up an application
run :: (LocalNode -> Process ()) -> IO (LocalNode, Transport)
run app = do
  eitherTrans <- newTransport

  case eitherTrans of
    Left err ->
      panic $ show err

    Right transport -> do
      node <- newLocalNode transport initRemoteTable
      _    <- runProcess node (app node)

      pure (node, transport)


-- | Sends to a named process
namedSend :: (Binary a, Typeable a) => String -> a -> Process ()
namedSend name msg = do
  mbPid <- whereis name

  case mbPid of
    Nothing ->
      say "process not registered"

    Just pid ->
       send pid msg

Testing

We'll start off by writing a custom HSpec hook to make a bridge between our application and our tests. Our hook will spin up an application and thread around a shared MVar and a LocalNode.

The MVar serves two purposes; it will be used to communicate state and act as locking mechanism (takeMVar blocks until it's full). Whilst the LocalNode will allow us to spin up adhoc processes when we need them.

The functions aroundApp and withApp are the first step in bridging these two worlds.

-- SPEC HELPERS


-- | Spins up and tears down app and passing along an mvar
aroundApp :: (MVar a -> LocalNode -> Process ())
          -> SpecWith (MVar a, LocalNode)
          -> Spec
aroundApp app =
  around $ withApp app


-- | Spins up application, closes it cleanly and passes along an mvar
withApp :: (MVar a -> LocalNode -> Process ())
        -> (((MVar a, LocalNode) -> IO ()) -> IO ())
withApp app action = do
  mvar              <- newEmptyMVar
  (node, transport) <- run $ app mvar
  finally (action (mvar, node)) (closeTransport transport)

The second step in bridging these worlds is defining a function that'll listen for messages that are sent to a process and put them in our MVar.

-- | Listens for messages and writes msg to an mvar
writer :: (Binary a, Show a, Typeable a) => MVar a -> Process ()
writer mvar = do
  msg <- expect
  liftIO $ putMVar mvar msg

Using these functions, we'll write our first test. It's important that we're confident our application spins up all of the relevant processes it needs to function correctly. We'll do this by starting our application, checking whether the process is registered and putting the result in our MVar.

-- SPECS


main :: IO ()
main = hspec $ do
  let
    double mvar node = do
      _ <- app node
      x <- whereis "client"
      y <- whereis "server"
      liftIO $ putMVar mvar [x, y]

  aroundApp double $
    describe "app" $ do
      it "should spin up every process" $ \(mvar, _) -> do
        mbPids <- takeMVar mvar
        any isNothing mbPids `shouldBe` False

From here we'll want to test that our processes communicate — i.e. send and receive appropriate messages — with one another as we expect. We do this by starting a client process and registering a server test double that'll listen for messages sent to it using the writer function.

  let
    double mvar node = do
      _ <- newProcess node "client" clientProcess
      _ <- newProcess node "server" $ writer mvar
      pure ()

  aroundApp double $
    describe "Ask ints" $ do
      it "should call the server process" $ \(mvar, node) -> do
        _ <- forkProcess node $
          namedSend "client" (Ask [1, 2, 3, 4])

        Calc _ ints <- takeMVar mvar
        ints `shouldBe` [1, 2, 3, 4]

Finally we'll want to test our server process calculates results correctly and sends them back to clients. We do this by starting our server process and sending a message to it from a test double client process.

  let
    double mvar node = do
      void $ newProcess node "server" serverProcess

  aroundApp double $
    describe "Result i" $ do
      it "should call the client process with the result" $ \(mvar, node) -> do
        _ <- forkProcess node $ do
          pid <- newProcess node "client" $ writer mvar
          namedSend "server" $ Calc pid [1, 2, 3, 4]

        Result i <- takeMVar mvar
        i `shouldBe` 10

Conclusion

The approach described in this post reflects some of my background in object-oriented programming. After all, spinning up processes and testing messages passed between them feels very similar to instantiating objects and doing the same thing.

There are obviously some shortcomings to the techniques described — the big one being that the type checker doesn't complain when you send an unknown message to a process. That said, the approach distributed-process makes you take is very consistent and makes it pleasant to write asynchronous applications.

Hopefully what I've written here offers some insight into how you might begin testing your distributed-process applications.

Parse Ruby Objects in Haskell

Mon, 24 Apr 2017 00:00:00 +0000

Parse Ruby Objects in Haskell

2017-04-24 haskell ruby

In 2015 I released my first Haskell project ruby-marshal. It's a package that uses the binary package to parse Ruby objects serialised with Marshal.dump. I wrote it in my spare time because I was curious to know whether I could devise a strategy to incrementally migrate legacy Ruby on Rails applications over to Haskell without the risk associated with a full rewrite.

My hypothesis was that if I could decrypt and de-serialise Rails sessions then I'd be able to piggyback on the Rails application's authentication mechanism. Not long after, I had the opportunity to use this package at work, and put this theory to the test, by writing a Haskell web application that shared sessions with Rails.

It has been running in production -- without any issue -- for almost two years.

Marshal

Ruby's Marshal library serialises Ruby objects to a bytestring e.g. dumping true results in [4, 8, 84] where 4 and 8 are the Marshal version number and true is represented as 84 or ASCII T.

% irb
irb(main):001:0>  x = Marshal.dump(true)
=> "\x04\bT"
irb(main):002:0> x.bytes
=> [4, 8, 84]

Compound objects, e.g. hash maps, can also be serialised using Marshal.dump. This might explain why it was used as the default cookie serialiser in Rails until version 4.1, after which JSON serialisation became the default.

% irb
irb(main):001:0>  x = Marshal.dump("session_id" => "ba0844151d")
=> "\x04\b{\x06I\"\x0Fsession_id\x06:\x06ETI\"\x0Fba0844151d\x06;\x00T"
irb(main):002:0> x.bytes
=> [4, 8, 123, 6, 73, 34, 15, 115, 101, 115, 115, 105, 111, 110, 95, 105, 100, 6, 58, 6, 69, 84, 73, 34, 15, 98, 97, 48, 56, 52, 52, 49, 53, 49, 100, 6, 59, 0, 84]

More information about the Marshal.dump binary format can be found in a series of blog posts by @jakegoulding or by reviewing the ruby-marshal source code.

Design

The ruby-marshal package allows us to transform this binary format into Haskell values and follows a pattern you'll see elsewhere in the Haskell ecosystem. It consists of:

An abstract syntax tree (AST) that represents Ruby objects.
A collection of parser combinators to transform the Marshal binary representation into an AST.
A custom monad to enrich the underlying Get monad with additional effects.

AST

The Ruby AST represents a subset of values that can be encoded by Marshal.dump.

data RubyObject
  -- Simple objects.
  = RNil
  | RBool Bool
  | RFixnum Int
  | RFloat Float
  | RString ByteString
  | RSymbol ByteString
  -- Compound objects.
  | RArray (Vector RubyObject)
  | RHash (Vector (RubyObject, RubyObject))
  | RIVar (RubyObject, RubyStringEncoding)
  -- Tag for unsupported objects e.g. Bignum.
  | Unsupported

This is a common pattern you'll see in other packages e.g. msgpack:Object and aeson:Value.

Parsers Combinators

Parsers are combined to build an AST e.g. parsing a raw bytestring is defined as follows.

getString :: Marshal BS.ByteString
getString =
  -- Label a parser to ensure label will be appended if the parse fails.
  marshalLabel "RawString" $ do
    -- Get the number of bytes in the bytestring.
    n <- getFixnum
    -- Get the number of bytes in the bytestring and lift it into the Marshal monad.
    liftMarshal $ getBytes n

It is then used by other parsing functions e.g. parsing a Ruby symbol.

getSymbol :: Marshal BS.ByteString
getSymbol = marshalLabel "Symbol" $ do
  -- Get bytestring.
  x <- getString
  -- Write symbol into the symbol cache.
  writeCache $ RSymbol x
  -- Return the bytestring.
  return x

Before being used in the top level parsing function that combines parsing functions and lifts values in to the Ruby AST.

getRubyObject :: Marshal RubyObject
getRubyObject =
  -- Make sure we're using a supported Marshal version, throw away the result and recursively parse our bytestring.
  getMarshalVersion >> go
  where
    go :: Marshal RubyObject
    go = liftMarshal getWord8 >>= \case
           NilChar        -> return RNil
           TrueChar       -> return $ RBool True
           FalseChar      -> return $ RBool False
           FixnumChar     -> RFixnum <$> getFixnum
           FloatChar      -> RFloat <$> getFloat
           StringChar     -> RString <$> getString
           SymbolChar     -> RSymbol <$> getSymbol
           ObjectLinkChar -> RIVar <$> getObjectLink
           SymlinkChar    -> RSymbol <$> getSymlink
           ArrayChar      -> RArray <$> getArray go
           HashChar       -> RHash <$> getHash go go
           IVarChar       -> RIVar <$> getIVar go
           _              -> return Unsupported

Marshal Monad

A quirk of the Marshal format is that it saves space by encoding repeated objects as indexes into a symbol cache and an object cache. We use StateT to keep track of these during de-serialisation and enrich the underlying Get monad by creating a custom monad.

newtype Marshal a = Marshal { runMarshal :: StateT Cache Get a }
  deriving (Functor, Applicative, Monad, MonadState Cache)

This allows us to write to and read from our cache during parsing without having to manually thread state through our parsing functions.

Examples

File IO

Let's take a simple example of a Ruby string, serialise it and dump it to the file system using irb.

% irb
irb(main):001:0>  x = "hello haskell"
=> "hello haskell"
irb(main):002:0> y = Marshal.dump(x)
=> "\x04\bI\"\x12hello haskell\x06:\x06ET"
irb(main):003:0> File.open("example.bin", "w") { |z| z.write(y) }

Switching over to Haskell we set up our imports.

import Data.ByteString (ByteString)
import Data.Ruby.Marshal
import qualified Data.ByteString as BS
import qualified Data.ByteString.Char8 as Char8

Define a function to read our example from the file system.

readExample :: IO ByteString
readExample = BS.readFile "example.bin"

Define a function that uses the Rubyable typeclass to convert a RubyObject to a more convenient representation.

toString :: RubyObject -> Maybe (ByteString, RubyStringEncoding)
toString rubyObject = fromRuby rubyObject

Before putting it all together to print the Ruby string to the console.

main :: IO ()
main = do
  -- Read example.bin.
  example <- readExample

  -- Decode using ruby-marshal and maybe convert the result to a bytestring.
  case decode example >>= toString of

    -- Handle the case when serialisation fails.
    Nothing ->
      putStrLn "Oops, something went wrong..."

    -- Throw away the encoding information and print the Ruby string to the console.
    Just (string, _) ->
      Char8.putStrLn string

Memcache

Let's take another example of de-serialising Ruby objects stored in memcache using the the dalli gem.

% irb
irb(main):001:0>  require "dalli"
=> true
irb(main):002:0> dc = Dalli::Client.new("localhost:11211")
=> #<Dalli::Client:0x007fe4948b7358 @servers=["localhost:11211"], @options={}, @ring=nil>
irb(main):003:0> dc.set("str", "hello haskell")

We'll reuse our existing Haskell code but add another import.

import qualified Database.Memcache.Client as M

Define a function that creates a new memcache client.

createMemcacheClient :: IO M.Client
createMemcacheClient =
  M.newClient [M.ServerSpec "localhost" 11211 M.NoAuth] M.def

Before putting it all together to pull the value out of memcache and print the Ruby string to the console.

main :: IO ()
main = do
  -- Set up memcache client.
  mc <- createMemcacheClient

  -- Retrieve bytestring from memcache server.
  example <- M.get mc "str"

  -- Unpack result from memcache server.
  case example of
    Nothing ->
      putStrLn "Oops, key not found..."

    -- Pattern match to extract bytestring.
    Just (value, _, _) ->

      -- Decode using ruby-marshal and maybe convert the result to a bytestring.
      case decode value >>= toString of

        -- Handle the case when serialisation fails.
        Nothing ->
          putStrLn "Oops, something went wrong..."

        -- Throw away the encoding information and print the Ruby string to the console.
        Just (string, _) ->
          Char8.putStrLn string

Conclusion

By writing the ruby-marshal package, I was able to create a Haskell web application that coexisted with a Rails application. This approach has been a success at work and appears to be one way in which you could gradually migrate an existing web application written in Ruby over to Haskell without the risk associated with a full rewrite.

Writing PureScript Bindings for a jQuery Plugin

Fri, 06 Nov 2015 00:00:00 +0000

Writing PureScript Bindings for a jQuery Plugin

2015-11-06 javascript purescript

You've started a new project at work with a tight deadline. You're in charge of picking the stack and after assessing technologies best for the project you've decided to use PureScript on the front-end. One factor determining the success of the project is that it be delivered on time.

PureScript is the technology I chose for a new project at work and one of the ways it's helping my company deliver our project on time is by allowing us to leverage existing JavaScript libraries using its foreign function interface (FFI), whilst still allowing us to exploit strong typing. The experience of writing FFI bindings has been so positive that I wanted to share with you how you might write your own bindings for an existing jQuery plugin.

In this post, together we'll be writing bindings for a subset of the functionality of jQuery-steps: a lightweight wizard UI component written for jQuery. This won't be a deep dive into PureScript's FFI but will give you an insight into some of the practical considerations involved in using JavaScript code from PureScript. It assumes familiarity with JavaScript, PureScript and jQuery.

FFI

PureScript's FFI allows communication from PureScript code to JavaScript code. It's straight-forward and flexible, making it possible to write bindings for even the most involved JavaScript code. Provided, that is, that you understand the underlying runtime representation of what you're dealing with.

Configuration

As is typical for jQuery plugins, jQuery-steps takes a configuration object to tell it which DOM elements it should bind to. We begin by defining the types we'll need to safely represent the fields of this configuration object in PureScript.

module JQuery.Steps where

import Control.Monad.Eff
import Control.Monad.Eff.JQuery
import Control.Monad.Eff.Unsafe
import DOM
import Data.Foreign.Callback
import Data.Function
import Prelude

newtype HeaderTag = HeaderTag String
newtype BodyTag = BodyTag String
newtype CssClass = CssClass String

We use newtype wrappers to represent the values of the fields in our configuration object. We do this because newtypes have the same runtime representation as the underlying type, whilst being distinct from the perspective of the type checker. This makes newtypes an easy win for providing an extra layer of type safety when defining FFI bindings.

type Config = { headerTag :: HeaderTag
              , bodyTag   :: BodyTag
              , cssClass  :: CssClass
              }

We use a type synonym to represent our configuration rather than an algebraic data type (ADT). Much like a newtype, the type synonym's runtime representation will have a direct correspondence with what jQuery-steps expects, whereas an ADT's constructors would present an extra level of indirection that we'd have to handle in our JavaScript wrapper.

defaultConfig :: Config
defaultConfig = { headerTag: HeaderTag "h1"
                , bodyTag: BodyTag "div"
                , cssClass: CssClass "wizard"
                }

We define a default configuration object to let consumers of our bindings use record update syntax to override the defaults we've set.

var defaultConfig = {
      headerTag: "h1"
    , bodyTag: "div"
    , cssClass: "wizard"
}

Here's what defaultConfig will look like at runtime. You can see that despite adding an extra layer of type safety when working with our configuration object in PureScript, we don't pay any extra cost for the abstraction at runtime. Practically speaking, this means that we can pass our configuration object directly to jQuery-steps.

/*global exports, jQuery*/

"use strict";

// module JQuery.Steps

exports.steps = function (object) {
    return function (config) {
        return function () {
            return jQuery(object).steps(config);
        };
    };
};

We define our export function that wraps jQuery-steps by providing a curried interface. It's worth noting that we're using the FFI style introduced in PureScript 0.7.

foreign import steps :: forall eff. JQuery -> Config -> Eff (dom :: DOM | eff) Unit

And, finally, we write a foreign import declaration and we're done. It's really as simple as that. Well, almost.

Callbacks and Dishonesty

Binding to DOM elements is all well and good but very soon you'll want to add dynamic behaviour such as client-side validation. This means delving into into the unprincipled world of JavaScript callbacks and exposing the plugin's event handler API.

type Config = { headerTag   :: HeaderTag
              , bodyTag     :: BodyTag
              , cssClass    :: CssClass
              , onFinished  :: Callback2 JQueryEvent Int Unit
              , onFinishing :: Fn3 JQueryEvent Int Int Boolean
              }

We begin by extending Config to include two new fields: onFinished and onFinishing. The type constructor Callback2 comes from purescript-foreign-callbacks and represents an effectful computation that takes two arguments, whilst the type constructor Fn3 comes from purescript-functions and represents a pure function that takes three arguments.

Something seems a bit odd here - why is onFinished effectful, whilst onFinishing is pure? Well, the truth is that they are both effectful, but since JavaScript offers no means of distinguishing between effectful and pure computations, we need to circumvent the type checker to ensure functions we pass to onFinishing have the correct runtime representation.

mkFinishing :: forall a. (JQueryEvent -> Int -> Int -> Eff a Boolean)
             -> Fn3 JQueryEvent Int Int Boolean
mkFinishing f = mkFn3 \x y z -> runPure <<< unsafeInterleaveEff $ f x y z

mkFinishing transforms our function into an appropriate runtime representation by uncurrying it and by unsafely pulling the inner value out of the Eff constructor. We uncurry our function because JavaScript functions aren't curried by default and we pull the inner value out of Eff because jQuery-steps expects a callback that returns a boolean value.

If at this point you're feeling a little uncomfortable, that's OK. Once a value is inside Eff you shouldn't really be pulling it back out without a very good reason. But we're dealing with JavaScript code here and managing the impedance mismatch, so consumers of our bindings don't have to, is probably reason enough. In any case, the callback function wouldn't be very useful if it was pure!

mkFinished :: forall a. (JQueryEvent -> Int -> Eff a Unit)
              -> Callback2 JQueryEvent Int Unit
mkFinished f = callback2 f

mkFinished, on the other hand, is much simpler. It uses callback2 to take our effectful computation of two arguments and transforms it into an appropriate runtime representation for jQuery-steps.

defaultConfig :: Config
defaultConfig = { headerTag: HeaderTag "h1"
                , bodyTag: BodyTag "div"
                , cssClass: CssClass "wizard"
                , onFinished: mkFinished \_ _ -> return unit
                , onFinishing: mkFinishing \_ _ _ -> return true
                }

And, finally, here's what the resulting defaultConfig looks like. Now we're really done. That wasn't so bad, was it?

Conclusion

PureScript lets us easily reuse existing JavaScript libraries with help from its FFI. Being able to reuse code can help us prototype more rapidly by allowing us to stand on the shoulders of giants. Combining this property with a strongly typed language can help us when that rapid prototype needs to make its way into a production setting.

Using Loggly On Elastic Beanstalk

Sun, 19 Jul 2015 00:00:00 +0000

Using Loggly On Elastic Beanstalk

2015-07-19 ops

This week I had to configure a Rails app running on Elastic Beanstalk to use Loggly. Since I don't want anyone to suffer the Elastic Beanstalk documentation more than is necessary, I've written this short tutorial should you find yourself tasked with the same challenge. It assumes you already have a Rails app running on Elastic Beanstalk and a general understanding of Rails apps.

Configure App

Add the syslogger gem to your Gemfile.

# Gemfile
gem 'syslogger', '~> 1.6.0'

In config/environments/production.rb configure your app to use Syslogger for logging. Replace "your-app-name" with, you guessed it, the name of your application. This allows you to easily see logs from your app in the Loggly web interface using source groups, which is especially useful if you're using Loggly to aggregate logs from multiple apps.

# config/environments/production.rb
config.logger = Syslogger.new("your-app-name", Syslog::LOG_PID, Syslog::LOG_LOCAL7)

Create Loggly Account

Go sign up for a Loggly account and create a customer token.

Configure Container Environment

Go to the Amazon AWS Console and add your Loggly account name and customer token to your Elastic Beanstalk app's environment using the LOGGLY_ACCOUNT and LOGGLY_AUTH property names.

Customise Container

Add the following configuration to loggly.config in the .ebextensions directory in the root of your app.

# .ebextensions/loggly.config
files:
  "/tmp/loggly_config.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/bin/bash
      su --command="python /tmp/configure-syslog.py setup --auth $LOGGLY_AUTH --account $LOGGLY_ACCOUNT --yes"

container_commands:
  01_loggly_dl:
    command:
      wget -q -O /tmp/configure-syslog.py https://www.loggly.com/install/configure-syslog.py
  02_loggly_config:
    command: /tmp/loggly_config.sh

This YAML file downloads Loggly's own syslog configuration script and sets up your container to send your system logs to Loggly. You can audit this script at loggly/install-script.

Wrap Up

Re-deploy your app to Elastic Beanstalk and watch as your logs begin appearing in Loggly.

Philip: music, code and stuff.

Notes on a 1987 Sampler's Firmware

Notes on a 1987 Sampler's Firmware

Tools for a 1987 Sampler

Tools for a 1987 Sampler

Notes on a 1987 Sampler

Notes on a 1987 Sampler

Testing distributed-process Apps Using Hspec

Testing distributed-process Apps Using Hspec

Application

Testing

Conclusion

Parse Ruby Objects in Haskell

Parse Ruby Objects in Haskell

Marshal

Design

AST

Parsers Combinators

Marshal Monad

Examples

File IO

Memcache

Conclusion

Links

Writing PureScript Bindings for a jQuery Plugin

Writing PureScript Bindings for a jQuery Plugin

FFI

Configuration

Callbacks and Dishonesty

Conclusion

Using Loggly On Elastic Beanstalk

Using Loggly On Elastic Beanstalk

Configure App

Create Loggly Account

Configure Container Environment

Customise Container

Wrap Up