How I delivered photo background manipulation app based on MMS

kevinhq.com - Long long ago, someone asked me to help develop MMS based app. The app will receive photos through MMS, replace the background with one of the chosen backgrounds, then send the result back. It seems like a simple app, but it’s not that simple. It takes about 40 hours of work to deliver the app.

The flow

At that time, there were several background photos. Each of the background photos maps to a phone number. Users will choose which background photo they want to use as a replacement for theirs.

Here’s the illustration

• Background A is on number +1 NXX NXX-XXX0
• Background B is on number +1 NXX NXX-XXX1
• Background C is on number +1 NXX NXX-XXX2

Let’s say we want to replace our photo background with Background C. To do that, we send our photo through MMS to this number +1 NXX NXX-XXX2. Then, the app will send us another MMS with the result where the background is now Background C. The same goes for the other two.

The tools

There are some tools I use to deliver such an app.

• I use Ruby as the programming language.
• For the server, I use the Heroku
• As the phone number provider and MMS processor, I choose Twilio and its API
• For the background photo removal, I use the remove.bg tool
• As the image processor, I use the ImageMagick library.

Here’s why I choose those tools: I need to integrate the app with another existing app that’s already running on Heroku.

The challenge

The first is removing the background accurately from the photo before someone suggests that tool. I figure out ImageMagick can’t remove accurately. The problem is it doesn’t delete cleanly. There’s always a leftover part of the background.

Another challenge is the processing time. Image processing can take some time. Yet, the app must send the result quickly. Here’s why it needs more time:

• There are several API calls to both Twilio and remove.bg tool, back and forth.
• It needs to parse the photo, scale its size if necessary, and merge it with the chosen background.
• When there are many concurrent requests, it should handle every image and the result correctly. It means sending the result to the correct sender’s phone number.

How do I test it

While in development, I test the app by creating several URLs. Each URL will send one test photo. So, I can send the test photo by opening each URL on the browser. After that, I can open the Twilio web console to see the result.

Verdict

Although the app has no user interface, it has its development challenges. To me, it’s as complicated as the web application you can open on a browser.