top of page
Writer's pictureStevan Dedovic

Reverse Zoology

This post is republished from it's original on my personal site, here: dedovic.com/writings/reverse-zoology.

 

Update March, 2022: Ellie and I will be participating in the Bright Moments Berlin event in April! Minting will be done in-person and via surrogate. If you’re able to, come out and support us! More info at: brightmoments.io/cryptoberlin

 

A few months ago I was introduced to Ellie Pritts, an outstanding artist out of L.A. who is popular in the NFT and digital art community. She was seeking a technical partner to work on a project, and I needed someone who can navigate art as a business. Our goals and personalities lined up pretty well, so, we decided to partner on what has now become the Reverse Zoology project.



It’s been a few months since we first chatted, and a lot of progress has been made. My goal here is to talk through some technical details, how the art works, and the numerous pitfalls I encountered along the way. I imagine there is more to discuss than can be done in a single post, but I have to begin somewhere.


An Overview

Reverse Zoology is an evolution of frogeforms with each piece created by the same process, parameterized.

For a single artwork, it looks something like this:

  1. Select 3 source images from a collection provided by Ellie

  2. Visually morph between the images, slowly changing

  3. Warp the outcome from step 2, making it wavy or distorted

  4. Composite the warped video from step 3 onto a background video

Ellie was responsible for generating the source images (Step 1) using a complex, psychedelic analog process, GANs, and much more! But that’s beyond the scope of my work and you can read about it in this Twitter thread.


Morphing

To visually morph between the images, I attempt to replicate an effect known as a Morph Cut or Smooth Cut.


The Morph Cut is a video transition effect that comes as a stock effect in many video editing softwares, such as Adobe Premier Pro and DaVinci Resolve. It is typically used to remove small sections of interviews, for example between unwanted pauses or stutter. In Adobe Premier Pro, according to their documentation, this is achieved using a combination of “face tracking and optical flow interpolation to create a seamless transition” between sections of video. DaVinci Resolve takes a similar approach, using optical flow processing, to perform the morphing effect. This transition becomes interesting when stretching it over the recommended range, of a few frames, to something much longer, like 10 or more seconds.



But how do these effects actually work? To answer that, we first need to look at a video morphing effect. My favorite use of morphing, and one of the earliest, come the amazing sequence at the end of Michael Jackson’s Black or White music video. See it happen around the 5:27 mark.



To achieve the face morphing effect, a transformation between two faces is performed where important features (eyes, nose, mouth, etc.) are moved and stretched from one person to the other. Next, the colors of individual pixels must be mixed to seamlessly transition over the duration of the effect.


A Custom Morphing Effect

This is where optical flow processing comes into play. Optical flow is a description of the motion of high-level features between two frames of video. For example, imagine a video of a basketball. The optical flow between two consecutive frames should tell us the ball is moving down while the background is static. Maybe a bird is somewhere in the shot, and that has its own motion. These are all high-level features, not individual pixels or edges.


Using an optical flow algorithm should allow me to predict the motion of things between two frames of video, or, if I feed it two of Ellie’s GAN animals, the optical flow between the two may yield some interesting results. The hope is mapping visually similar features and morphing it in an interesting way. Luckily for me, OpenCV has a handful of (mostly) out-of-the-box optical flow processing algorithms. The output of these algorithms is a vector field, telling me where the algorithms think regions of the first image move to line up on the second image.

The morphing effect now looks something like this:

  1. Get two source images, A and B

  2. Calculate the optical flow fields

    1. Flow AB = Image A → Image B

    2. Flow BA = Image B → Image A

  3. For every pixel in Image A

    1. Move it based on Flow AB

    2. Change the color, mixing between Image A and Image B

  4. Repeat for every pixel in Image B, using Flow BA

So… what does that look like?



Awesome! It’s not the same as Premier Pro but its unique and I can parameterize the effect to fit the content.

Warping

Now that morphing is out of the way, the next step is to warp the video, starting by defining warping. Warping is a distortion effect on a video. Something like the effect of ripples of water or of looking through a thick piece of glass. It means the light coming into my eyes isn’t traveling a straight path, its bent, bounced, blocked along the way. So, why does that happen, in the case of thick, abnormal glass?

Light refracts. If a ray of light travels through a medium of different density or composition as air, it will not travel a straight path. That’s why a straw or pen looks weird in a glass of water, like the image below.



Water causes refraction in what we see because of the physics of light. It follows well-known rules, and, more importantly, it follows well documented rules. Rules that computer graphics folk have already figured out and implemented - meaning I can simulate light refraction without having to reinvent too much.

So let’s describe a system for warping an image. We have:

  • a coordinate system. X is left-right, Y is up-down, Z is in-out

  • a camera, somewhere on the Z axis, looking in

  • an image sitting on the XY plane, looking out to the camera

  • every pixel of my screen is mapped 1:1 to a pixel on the image

That’s easy, I am displaying an image without modification. But now, let’s add a glass material. This will be a simple differentiable height map, meaning I can get the height of the glass and its normal, i.e., the 3D angle of the surface, by specifying an X and Y coordinate.

Now, updating the warping system from below, we must:

  1. calculate how far a ray of light travels before it hits the glass

  2. find the angle between the ray of light and the surface of the glass

  3. use Snell’s law to determine the angle of refraction based on the properties of the glass and the air

  4. measure how long the ray of light will travel, in its new direction, before hitting the image

  5. determine the pixel which is hit by the refracted ray of light

Now we have a physically based warping effect!



Putting it all Together


I will cover this section in more detail in a follow up post. Feel free to contact me if there is something you want me to discuss or detail!


Tools, Languages, and Frameworks

I chose to develop the main processing application in Rust. I wanted a performant language with fine-grained control over parallel programming, which excluded Python, Javascript, and Ruby. Not only that, but I also needed access to many low-level libraries and APIs, opting not to use a JVM language. Rust is fun and I find C++ to be verbose and dangerous.


For managing dependencies, the Nix package manager seemed like a great fit. I’m writing code on my Mac and need to have my partner’s laptop working without issue. Installing OpenCV, GStreamer, Rust, etc. on a phone call was a non-option. But a simple git clone && nix-shell could work. I did have some issues making the main codebase function on both Mac and Linux, and I don’t know if I’ll ever figure that out. Nix is simply not friendly when it comes to graphics applications.


For optical flow processing, the heavy lifting is done by OpenCV. For high-performance morphing, warping, and compositing, I wrote OpenCL kernels. I started the project writing OpenGL shaders but realized this isn’t ideal. I don’t need any OpenGL context since I will be writing compute shaders, I’m essentially bastardizing the framework for my use case. OpenCL is smoother sailing.


For the UI, I decided on Nannou and Conrod. This was a mistake, but I’m living with it. Nannou is a good project, but not well suited for building UIs that are decoupled from some underlying computation. What makes it even more difficult the Conrod project - it is not maintained and a headache to use. If I could rewrite everything from scratch, I would stick to a client-server model, possibly communicating over TCP or HTTP.


Closing Remarks


Overall, this has been one of the biggest projects I’ve embarked on. I made numerous mistakes and learned many hard lessons. It is the single biggest Rust project I’ve worked on, bumping against many rough edges of the language. I have to say, I’m a Rust evangelist now. And a Nix evangelist too.


The next few topics I think I’ll cover are:

  • Glass warping, in detail, with some actual code

  • OpenCL kernels and performance considerations

  • My GStreamer based video frame grabber

  • Thoughts on using Rust, OpenCV, OpenCL, and Nix together to create a (mostly) cross-os development environment


 

Stevan Dedovic is a software engineer and visual artist based out of Chicago. His focus is on computation expressionism, high performance software, and sharing what he learns along the way. You can follow his work through Instagram, @mr_praline, or on his personal site, dedovic.com.


499 views130 comments

Recent Posts

See All

130 Comments


NEERAJ MISHRA
NEERAJ MISHRA
12 minutes ago

If you’re looking for fast publication and low fees, here’s a summary of your options based on the journals and indexing services you’re considering:

IJSET (International Journal of Scientific Engineering and Technology):

IJSRET (International Journal of Scientific Research and Engineering Trends):

 IJRTSSH (International Journal for Research Trends in Social Science & Humanities):

ABCD Index:

  • Role: The ABCD Index is an indexing platform, not a journal. It can provide visibility for your published paper if your chosen journal is indexed here. 

If you're looking for journals with low publication fees, here are some recommendations across disciplines, including the ones you're interested in: 


Like

Selina Soil
Selina Soil
4 days ago

Are you struggling with complex IT concepts or tight assignment deadlines? Our Information Technology Assignment Help service is designed to provide expert guidance and top-notch solutions for students pursuing IT courses. Whether it's programming, database management, networking, cybersecurity, or cloud computing, our team of experienced professionals ensures well-researched, accurate, and plagiarism-free assignments.

Why Choose Us?

  • Expert Assistance: Skilled IT professionals with academic expertise.

  • Timely Delivery: Submit your assignments before the deadline.

  • Affordable Pricing: High-quality services within your budget.

  • 24/7 Support: Round-the-clock assistance for all your queries.

Visit Here

Get the grades you deserve with our comprehensive and reliable IT assignment help. Visit here to know more and transform your academic journey today!


Like

dixim47802
Nov 28

Professional Nursing Writing Tailored to Your Unique Needs

In the healthcare field, nursing is one of the most demanding professions, requiring both practical and theoretical knowledge to provide optimal patient care. Nursing students face the challenge of balancing clinical practice with academic requirements, which can often lead to stress, burnout, and difficulty in completing assignments. The ability to write well-researched, clear, and insightful nursing papers is an essential skill for success in nursing programs. However, given the heavy workload and rigorous study schedules, many students turn to professional nursing writing services for assistance. These services offer customized, high-quality papers that are tailored to meet each student's specific needs, ensuring that they meet academic standards while allowing students to focus on…

Like

NEERAJ MISHRA
NEERAJ MISHRA
Nov 21

If you're looking for journals that offer fast-track publishing for research papers, here are some tips and resources:

1. Fast-Track Journals

Many journals provide expedited review and publication processes for authors who need their work published quickly. These options often involve additional fees or strict manuscript requirements. Below are some suggestions:

Actionable Steps

  • Identify Your Target Journal: Select a reputable journal in your field with a fast publication process.

  • Contact Editors: Email the editor to inquire about expedited options if they're not explicitly listed.

  • Ensure Readiness: Ensure your manuscript is error-free and formatted according to the journal's…


Like

seheti5548
Nov 18

https://forumweb.hosting/members/cafosi5225.31666/#about

https://www.miguelassis.com/group/mysite-231-group/discussion/eff296b2-d84a-42c2-b5c2-d218edff7280

https://trabajo.merca20.com/author/letog51430/

http://www.biznas.com/Biz-postsm339389_-Emergency-Nursing--Challenges-and-Rewards-in-Critical-Care.aspx#post339389

https://freshsites.download/socialwow/upload/files/2024/07/fTIQ1v5rYoBWJ2cQDVlK_30_7132e3503228007a640fb83f710b3558_file.pdf

https://mellrakforum.hu/forums/users/katovoh367/

https://vadea.viaafrika.com/forums/users/nogeh20722/

https://lyfepal.com/forums/thread/3099/Pediatric-Nursing-Specialized-Care-for-Young-Patients

https://bresdel.com/blogs/626348/Mental-Health-Nursing-Addressing-the-Growing-Need

https://charismaticplanet.com/author/dayojo7766/

https://www.hanaromartonline.com/forum/main/comment/492439fe-7ae5-4908-82ab-d10590d2d327?postId=63f363a564ec0c0012294d19

https://news.bangboxonline.com/the-impact-of-nursing-writing-services-on-career-advancement

https://firstamendment.tv/read-blog/52733

https://www.bhimchat.com/1722352258140517_51438

https://www.seereadshare.com/author/dayojo7766/

https://www.design-buzz.com/author/dayojo7766/

https://www.telix.pl/profile/dayojo7766/

https://coolmomsrule.com/author/dayojo7766/

https://hikyou.jp/forums/users/habed89499/

https://www.pmimauritius.com/forum/main/comment/fe65553b-a596-4d98-9e00-c725cc632b73?postId=6426888fb5e645001266ca97

https://www.tatasechallenge.org/community/forums/users/dayojo7766/

https://animationpaper.com/forums/users/nogeh20722/

https://www.odeh.ps/wall/blogs/794/The-Future-of-Nursing-Writing-Services-Trends-and-Innovations

https://directory.womengrow.com/author/dayojo7766/

https://targetshooting.ca/forums/users/dayojo7766/

https://www.clearnotebooks.com/ja/authors/6879739

https://projectnursery.com/author/dayojo7766/

https://muabanthuenha.com/author/dayojo7766/

https://git.bolin.su.se/bolin/keuper-wild-2020/-/issues/1409

https://truckymods.io/user/256140

https://drivehud.com/forums/users/dayojo7766/

https://legalforms.ng/ask-a-lawyer/profile/dayojo7766/

http://mehfeel.net/mehfeel/blogs/post/29225

https://bavave.com/wp-content/uploads/2024/07/Untitled-document-10.pdf

https://www.hituponviews.com/wp-content/uploads/2024/07/Untitled-document-11-1.pdf

https://www.wrkz.work/thread-3-post-4072.html#pid4072

https://git.ctdo.de/interfisch/hydroponic-controller/issues/7#issue-1724

https://photozou.jp/user/top/3369193

https://forum.fractalfuture.net/t/wann-werden-ki-roboter-schlau-er-als-der-mensch/2085/897?u=dayojo7766

https://bustedwallet.com/author/dayojo7766/

http://demo.advised360.com/read-blog/164468

https://kambadyami.net/upload/files/2024/07/4vuolE88UPMm67x1qCxR_31_290e8832291135fecf23580239e585aa_file.pdf

https://www.pnc-contact.com/forums/users/dayojo7766

https://phoenixsunsclub.com/read-blog/5864

https://www.surajmani.in/forum/main/comment/cc2ae09a-29a5-433e-88c9-3a8c238eb44f?postId=630821b42046ce00128dccc6

https://www.blackswancountryclub.com/forum/main/comment/105d92bb-ad6b-4124-8165-8ad65d9a9e50?postId=66aa65e2a23d670011b9c483

https://gitlab.com/-/snippets/3733747

https://git.project-hobbit.eu/-/snippets/26034

https://www.frankentoon.com/profile/katovoh367/profile

http://logebiv306.stranky1.cz/nova-stranka-328123/

https://www.mmicc.org/forum/main/comment/b0105693-5f87-4aca-a3bf-f22a8f3375dd?postId=6352717209d83d00128a393f

https://www.lidinterior.com/interioworld/main/comment/8fe18f52-2c3e-46eb-94e9-119cdcd77b2d?postId=6155856419c1e90016758949

https://www.cloudtenpictures.com/forum/main/comment/8ac0b444-baad-47f2-9d76-437b796114a2?postId=6446c6a0a7cf760012bdee20

https://www.broadwaychurchkc.org/forum/main/comment/84790e01-3c84-4cd5-8fca-69268461735a?postId=63ca79543421a1001218670e

https://www.exoltech.ps/blogs/post/61016

https://www.academicexperts.org/discussions/20134/?c=186446

https://tutorsonspot.com/questions/from-theory-to-practice-applying-academic-knowledge-with-the-help-of-nursing-writing-services-8xsz/

https://www.vgnetwork.com/members/vavohik964/profile/

https://linguacop.eu/blog/the-economic-impact-of-nursing-writing-services-on-healthcare-institutions/

https://forumforex.id/forum/forumforexid/rule-informasi/59314-effective-communication-strategies-in-nursing-writing-services

https://www.ictdemy.com/assembly-forum/assignment-help-company-in-australia-case-study-help-62c4195d2ea8d?page=1#goto10495

https://heroes.app/blogs/432295/The-Benefits-of-Editing-and-Proofreading-Services-for-Nursing-Papers

https://empowering-nursing-profe-pqddsaq.gamma.site/

https://www.debwan.com/blogs/563649/How-Nursing-Writing-Services-Can-Help-with-Complex-Case-Studies

https://www.mindspec.org/author/dayojo7766/

http://socialnetwork.linkz.us/blogs/78327/From-Theory-to-Practice-How-Nursing-Paper-Writing-Services-Aid

https://www.inventoridigiochi.it/membri/civalor491/

https://airsoftc3.com/article/8118/The-Future-of-Nursing-Writing-Services-Trends-and-Predictions

https://cafosi5225.diary.ru/

https://www.fundable.com/wayoli2661-wayoli2661

https://myjoye.com/forums/topic/11893/professional-development-through-nursing-writing-services/view/post_id/309085

https://www.saprec.org/forum/main/comment/d7c8c90a-cb22-44ea-b5f5-683caeaf18c5?postId=5fa91c1af6df54002d9c86e2

https://dawlish.com/user/details/22153

https://forum.gettinglost.ca/user/katovoh367

https://www.forum.motokobiety.pl/wiad602333.html#602333

https://www.gemresearchuk.com/forum/main/comment/f3ae2491-d7e5-423a-bf0e-6b7d9de07c96?postId=643305580c258a00135f4d14

https://thecaptivestory.com/community/profile/civalor491/

https://war-lords.net/forum/thread-937-post-138481.html#pid138481

http://share.pkbigdata.com/orvillegoldsch/replica-bags1986/issues/1

https://ivalor491itivelycom.livepositively.com/overcoming-writer-s-block-in-nursing-writing/new=1

https://writefreely.ethibox.fr/civalor491/span-style-text-decoration-underline-em-strong-andnbsp-using

https://www.whizolosophy.com/category/activity-fitness-sport/article-column/writing-reflections-in-nursing-a-guide-for-students

https://notepadonline.co/share/jy51f91722685508

https://forum.goldenear.com/forum/user/14708-civalor491

https://dictanote.co/n/1046579/

https://www.themoviedb.org/u/civalor491

https://www.realestatedepot.com/forums/discussion/general/join-the-best-aws-cloud-course-in-pune-for-career-growth

https://www.robertehall.com/profile/logebiv306/profile

https://aboutnursepractitionerjobs.com/author/habed89499/

https://www.keiteq.org/forum/main/comment/e075f12c-50fb-4211-86af-2bd28d770055?postId=5f7d862f4cbeaa00b9ba04b2

https://preview.shorthand.com/cAUmQhCY21vxFCsc/responsive/desktop

https://v8.next.forgejo.org/civalor491/treert/issues/1#issue-495

https://gratitudehabitat.com/forums/topic/have-you-used-authors-or-services-to-write-literature-reviews/page/8/#post-1329428

https://amazingradio.com/profile/civalor491

https://www.weddingbee.com/members/civalor491/

https://www.victoria-restaurant.net/wall/blogs/1255/Ethical-Writing-in-Nursing-Guidelines-and-Responsibilities

https://www.aegeanonline.edu.gr/wall/blogs/874/Writing-Tools-and-Resources-for-Nursing-Professionals

https://www.cowcow.com/user/civalor491-3203300/profile

https://www.cdt.cl/user/civalor491civalor491/

https://www.lumberjocks.com/members/civalor491.394675/#about

https://staging.partners.org/myapnea.org/members/civalor491/posts

Like
bottom of page