Adding 48-bit support for pix2pix

I’ve been experimenting with pix2pix and I’m literally loving it. This is such a great tool to create various image to image mappings.

I’ve currently made some modifications to get it to support 48-bit PNG support. This allows to represent 65536 values instead of the usual 256 values allowing for a higher resolution of data to be stored in an image. The network doesn’t really care because it all gets converted to float values anyway.

Hope this helps someone. I’ve forked the repo at: https://github.com/tharindu-mathew/pytorch-CycleGAN-and-pix2pix

environment.yml for vid2vid for easier setup

Vid2vid seems to be a promising technique for video synthesis using GANs, as of 2019, which is similar in spirit to it’s image counterpart, pix2pix. When setting this up, I found there are some additional requirements needed, than what’s stated at: https://github.com/NVIDIA/vid2vid.

I created my own environment.yml to make this setup easier (pytorch is specifically for CUDA 9.1, CUDNN 7.1.2, but you can edit the yml file and swap out this version based on the available pytorch versions, and it should work).

The gist is available at: https://gist.github.com/tharindu-mathew/7f79433d92d884fa662f6ccf023537e1

How to improve your shadows – Understanding the light projection matrix

When using shadow mapping, the resolution of your shadow buffer is lower than your default color buffer resulting in a low resolution shadow. But, this effect can be somewhat mitigated by programatically computing your light projection matrix so it covers the minimal volume possible.

In the above images, the buffer of the shadow resolution is the same. But, the bounds use for the computation of the light matrix (light_projection_matrix * light_view_matrix) differs. For the high res shadows on the right, the light projection matrix is controlled by the bounds of the view frustrum. In terms of a directional light, this will be an orthographic matrix, and the bounds of the light be the bounds of the viewing frustrum (which can be perspective or orthographic). This makes the resolution of the shadows far more high-res as only a smaller portion of your scene is considered. So, if you choose a reasonable resolution and use a bounds-aware light projection matrix you get good use resolution of your shadows. This can vary for level of detail that you shoot for based on the zoom of your camera.

Binding to framebuffer 0 may cause a blank screen


Today, I encountered an interesting issue that took away a few hours of my (not so) precious life, to debug and understand. I was implementing some basic shadow mapping, which requires you to create a new render target (meaning you need to render to a separate buffer other than the screen). So, we have to switch back and forth between framebuffers. Most OpenGL tutorials out there will simply ask you to bind back to the default framebuffer 0.

Here’s a code excerpt from learnopengl.com site (as of 11/16/2018) from their article on shadow mapping (I love this site, and this in no way a criticism of their content, just using it to point to a probably bug). https://learnopengl.com/Advanced-Lighting/Shadows/Shadow-Mapping

// 1. first render to depth map
glViewport(0, 0, SHADOW_WIDTH, SHADOW_HEIGHT);
glBindFramebuffer(GL_FRAMEBUFFER, depthMapFBO);
glClear(GL_DEPTH_BUFFER_BIT);
ConfigureShaderAndMatrices();
RenderScene();
glBindFramebuffer(GL_FRAMEBUFFER, 0);
// 2. then render scene as normal with shadow mapping (using depth map)
glViewport(0, 0, SCR_WIDTH, SCR_HEIGHT);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
ConfigureShaderAndMatrices();
glBindTexture(GL_TEXTURE_2D, depthMap);
RenderScene();

Binding back to framebuffer 0, as mentioned here, simply output a blank screen for me. And I went through a period of commenting out all my rendering code and enabling line by line (costing me half a day or more) to see what was going wrong. I found the culprit in the line: glBindFramebuffer(GL_FRAMEBUFFER, 0);. Then I suddenly got an idea and executed these lines to find my actual default framebuffer (after disabling any shadow mapping code and simply doing a simple setup for single pass rendering):


glGetIntegerv(GL_DRAW_FRAMEBUFFER_BINDING, &default_draw_fbo_);
glGetIntegerv(GL_READ_FRAMEBUFFER_BINDING, &default_read_fbo_);

And to my surprise, the answer was 3. With allocating a new framebuffer for shadow mapping, this went up to 4 (weird!). I’m using Qt 5.11 as my application framework, and I’m not sure whether this is a bug/feature or what (maybe they use framebuffer 0 to render their own stuff). But, it seems that the default framebuffer cannot be assumed to be 0.

So, if you’re experiencing a blank screen when trying to a render anything that causes you to switch between framebuffers, make sure you find out what exactly you’re default framebuffer ID is. Then, just switch back to this known number, and all will be well.


// also GL_FRAMEBUFFER is deprecated now, simply use
glBindFramebuffer(GL_DRAW_FRAMEBUFFER, default_draw_fbo_);

 

HTH.

A comparison with and without normal maps

I implemented a normal map into my tiny graphics engine (I’ve gotten by mostly with phong shading for my research work, so yeah I should do more graphics stuff I thought), and also to learn about tangent space. I saw firsthand the remarkable difference a normal map brings.

Map without normal map

without-normal-map

Map with normal map (same lighting as above)

with-normal-map

It’s pretty cool how much more detail a normal map brings into your models. Hope to integrate more features like this.

Loading Shininess in Materials when using Assimp

I was looking around on how to get the shininess (aka the specular coefficient, specular exponent) from a loaded model, when using the Assimp APIs (4.1.0). Unfortunately, nothing in the docs helped me and I had to look into some sample programs to get the answer. So, here’s how to do it:

// material index is the ID of the material you are interested in
	const aiMaterial* ai_material = scene.mMaterials[material_index];
float shininess;
if(AI_SUCCESS != aiGetMaterialFloat(ai_material, AI_MATKEY_SHININESS, &shininess))
    {
// if unsuccessful set a default
shininess = 20.f;
    }

Here’s an example to get some other property like specular color using Assimp’s structs.

aiColor4D spec_color;
 if(AI_SUCCESS != aiGetMaterialColor(ai_material, AI_MATKEY_COLOR_SPECULAR,
        &spec_color))
    {
        spec_color = aiColor4D(1.f, 1.f, 1.f, 1.f);
    }

HTH

Building Caffe on Windows with CUDA 9.0 for VS 2013, 2015 and 2017

Building Caffe on Windows 10 has been a journey (to put it lightly). Since I have a new gen gfx card (new for 2018), it does not support CUDA 8.0. So, all posts saying you need CUDA 8.0 are outdated (at least to me). With CUDA 9.0, Windows caffe does not compile.

The good news after a couple of days of trying I’ve figured out a workaround. The only boost version that supports CUDA 9.0 as of now is boost 1.65.1 and above. But, interestingly cmake breaks with boost 1.66.0. I know, welcome to the real-world versioning hell when it comes to actually building stuff.

So, if you got your windows source from: https://github.com/BVLC/caffe/tree/windows

You need to do a couple of stuff. First download and install boost 1.65.1 in some path. Let’s call this root directory my_boost_1_65_1 (typically C:\local\boost_1_65_1), and the library directory (which changes based upon which VS version you downloaded, typically C:\local\boost_1_65_1\lib64-msvc-14.0 for VS 2015). Yes, it sucks that MSVC version is 14.0 for VS 2015, but such is the life living in a Microsoft world.

I assume you have checked out the 1.0 version of windows caffe. Now, open the build_windows.cmd in the scripts directory, and modify the cmake command as follows (note the 3 lines referring to boost and boost paths):

cmake -G"!CMAKE_GENERATOR!" ^
-DBLAS=Open ^
-DCMAKE_BUILD_TYPE:STRING=%CMAKE_CONFIG% ^
-DBUILD_SHARED_LIBS:BOOL=%CMAKE_BUILD_SHARED_LIBS% ^
-DBUILD_python:BOOL=%BUILD_PYTHON% ^
-DBUILD_python_layer:BOOL=%BUILD_PYTHON_LAYER% ^
-DBUILD_matlab:BOOL=%BUILD_MATLAB% ^
-DCPU_ONLY:BOOL=%CPU_ONLY% ^
-DCOPY_PREREQUISITES:BOOL=1 ^
-DINSTALL_PREREQUISITES:BOOL=1 ^
-DUSE_NCCL:BOOL=!USE_NCCL! ^
-DCUDA_ARCH_NAME:STRING=%CUDA_ARCH_NAME% ^
-DBOOST_NO_BOOST_CMAKE=TRUE ^
-DBOOST_ROOT=C:/boost_1_65_1 ^
-DBOOST_LIBRARYDIR=C:/boost_1_65_1/lib64-msvc-14.0 ^
"%~dp0\.."

I do the following edit as well, so I know that boost version is correct or it will fail. So, edit the boost version in Dependencies.cmake (located in the cmake directory):


# ---[ Boost
find_package(Boost 1.65 REQUIRED COMPONENTS system thread filesystem)
list(APPEND Caffe_INCLUDE_DIRS PUBLIC ${Boost_INCLUDE_DIRS})
list(APPEND Caffe_DEFINITIONS PUBLIC -DBOOST_ALL_NO_LIB)
list(APPEND Caffe_LINKER_LIBS PUBLIC ${Boost_LIBRARIES})

And, by changing these two files, everything should be good. Now, execute scripts\build-windows.cmd and watch your build succeed.


l_prerequisites.vcxproj]
CUSTOMBUILD : warning : cannot resolve item 'api-ms-win-crt-heap-l1-1-0.dll' [D:\tools\caffe\build\tools\upgrade_solver_proto_text_install_prerequisites.vcxproj]
CUSTOMBUILD : -- warning : gp_resolved_file_type non-absolute file 'api-ms-win-crt-heap-l1-1-0.dll' returning type 'system' -- possibly incorrect [D:\tools\caffe\build\tools\upgrade_solver_proto_text_install_
prerequisites.vcxproj]
CUSTOMBUILD : warning : cannot resolve item 'api-ms-win-crt-stdio-l1-1-0.dll' [D:\tools\caffe\build\tools\upgrade_solver_proto_text_install_prerequisites.vcxproj]
CUSTOMBUILD : -- warning : gp_resolved_file_type non-absolute file 'api-ms-win-crt-stdio-l1-1-0.dll' returning type 'system' -- possibly incorrect [D:\tools\caffe\build\tools\upgrade_solver_proto_text_install
_prerequisites.vcxproj]
CUSTOMBUILD : warning : cannot resolve item 'api-ms-win-crt-convert-l1-1-0.dll' [D:\tools\caffe\build\tools\upgrade_solver_proto_text_install_prerequisites.vcxproj]
CUSTOMBUILD : -- warning : gp_resolved_file_type non-absolute file 'api-ms-win-crt-convert-l1-1-0.dll' returning type 'system' -- possibly incorrect [D:\tools\caffe\build\tools\upgrade_solver_proto_text_insta
ll_prerequisites.vcxproj]

6897 Warning(s)
0 Error(s)

Time Elapsed 00:14:44.49

Let me know in the comments if you run into more issues.

Cut down video size while maintaining quality

I believe the best way to do this is through ffmpeg (Windows build – http://ffmpeg.zeranoe.com/builds/, others –https://ffmpeg.org/).

Copy your input video (ex: in.mp4 to the ffmpeg bin folder). Open up the command line and navigate to the same bin folder and enter this command. Here out.mp4 is the output video.

ffmpeg -i in.mp4 -crf 20 out.mp4

It works for most videos extremely well. Saved me when my videos were over the size limit for research submissions.