Android Question Is it possible to process images in real time?

[irda]

Hello. I am new and I need to know which of the following are possible which make with B4A and not because they are supported by a mobile. It is an application that analyzes images in real time to show some information at the end of the process.
- Start the camera. It is possible using the library CameraEx.
- Access the camera preview: if the function Sub Camera1_preview (Data () As Byte) of the same library.
- Extract RGB values of each pixel vector Data ()?. I think this format Data Ybr.
-Loop through all the pixels for values. Do you think that will be too slow? Can you capture less resolution to go faster?

All this should make about 20 times per second. Is it too much effort for a phone?

I appreciate any help.

 

[thedesolatesoul]

- Extract RGB values of each pixel vector Data ()?. I think this format Data Ybr.
-Loop through all the pixels for values. Do you think that will be too slow? Can you capture less resolution to go faster?

All this should make about 20 times per second. Is it too much effort for a phone?

IMHO, this will be too slow. Firstly just getting the bitmap will be slow, then looping through all the pixels will be a slow operation.
20 fps is quite high with this approach.
You can decrease the resolution to make it faster. You can also try to do some bulk operations on the pixels to make it fast.

 

[Erel]

I also think that it will be too slow. What do you need to do with the RGB values?

 

[irda]

I need to detect and follow an item.
What do you mean when you are referring to bulk operations and how could I do them?
Thanks

 

[Erel]

You will need to use a framework such as OpenCV for that.

 

[mindcom]

Dear All,
I would like to process directly data() that comes from a _PictureTaken event (CameraEx library).
What is the format?
Can I make something like GetPixel directly on the data() array?
Many thx for your help.
Andrea

 

[Erel]

The default format is Jpeg. Seems like most devices only support this format.

 

[mindcom]

What I need is to use a function to extract color of some pixel in the image, just like GetPixel(x, y).
What I have to do now is to save the data() array to a file with cameraEx.SavePictureToFile(data, dir, filename). Than I have to open it again with LoadBitmap or LoadBitmapSample. And now I have a Bitmap object where I can use GetPixel.
No way to that process directly on data() array?

 

[Informatix]

What I need is to use a function to extract color of some pixel in the image, just like GetPixel(x, y).
What I have to do now is to save the data() array to a file with cameraEx.SavePictureToFile(data, dir, filename). Than I have to open it again with LoadBitmap or LoadBitmapSample. And now I have a Bitmap object where I can use GetPixel.
No way to that process directly on data() array?

Look at the GetPixels function of the ImageUtils class of my Accelerated Surface library.

 

[mindcom]

Look at the GetPixels function of the ImageUtils class of my Accelerated Surface library.

Informatix, I did not found a solution with your AS library.

I have found a simple way to analize preview data...

Sub Camera1_Preview (data() As Byte)
Dim dataJpg() As Byte, iSt As InputStream, img As Bitmap
dataJpg = camEx.PreviewImageToJpeg(data, 80)
iSt.InitializeFromBytesArray(dataJpg, 0, dataJpg.Length)
img.Initialize2(iSt)
...

I would like to do the same whan I take a picture at full size...
Any idea?

 

[Informatix]

Informatix, I did not found a solution with your AS library.

What do you mean? That you do not find the function in the library? Or it is not suited for your needs (so what are your needs because your initial request was to get the color of each pixel of an image)?

 

[mindcom]

What do you mean? That you do not find the function in the library? Or it is not suited for your needs (so what are your needs because your initial request was to get the color of each pixel of an image)?

I use without any problem the getpixel function. My problem is to get and android.graphics.Bitmap from the byte array data() in YUV format that you get from the _PictureTaken (Data() As Byte) event from Camera.
At the moment the only way I have found is to save the Data() byte array with the camera.SavePictureToFile(Data, folder_name, file_name) fuction, that generate a jpeg file. Than I have to open it again with LoadBitmap or LoadBitmapSample. So I obtain an android.graphics.Bitmap where I can use a getpixel function. But it is a very slow solution.

My question is:
1) is there a way to use a getpixel function directly on the Data() byte array?
or 2) is it possible to add to the Camera object a function like .PreviewImageToJpeg, able to process the full size image that comes from the _PictureTaken event?
or 3) any other solution to process directly data stream from the camera as fast as possible?

 

[Informatix]

I use without any problem the getpixel function. My problem is to get and android.graphics.Bitmap from the byte array data() in YUV format that you get from the _PictureTaken (Data() As Byte) event from Camera.
At the moment the only way I have found is to save the Data() byte array with the camera.SavePictureToFile(Data, folder_name, file_name) fuction, that generate a jpeg file. Than I have to open it again with LoadBitmap or LoadBitmapSample. So I obtain an android.graphics.Bitmap where I can use a getpixel function. But it is a very slow solution.

My question is:
1) is there a way to use a getpixel function directly on the Data() byte array?
or 2) is it possible to add to the Camera object a function like .PreviewImageToJpeg, able to process the full size image that comes from the _PictureTaken event?
or 3) any other solution to process directly data stream from the camera as fast as possible?

Ok I see. What result did you get when you pass your byte array to your Camera1_Preview sub (and pass it directly to InitializeFromBytesArray since the data are already in JPG format)?

Note that GetPixel and GetPixels are two different functions and I wouldn't use GetPixel to get all the pixels of an image because it's horribly slow.

 

[mindcom]

Ok I see. What result did you get when you pass your byte array to your Camera1_Preview sub (and pass it directly to InitializeFromBytesArray since the data are already in JPG format)?

Note that GetPixel and GetPixels are two different functions and I wouldn't use GetPixel to get all the pixels of an image because it's horribly slow.

The _Preview event is generated by the Camera, and what I have done in preview is to use the .PreviewImageToJpeg function to convert the Data() byte array from YUV format to JPG stream. Then I have initilized an InputStream with the .InitializeFromBytesArray function. Then I have initialized a normal bitmap with the .Initialize2 function. In my opinion too passages to access memory, but it works fine and make it possible to do some simple analisys on preview frames.

Sub Camera1_Preview (data() As Byte)
Dim dataJpg() As Byte, iSt As InputStream, img As Bitmap
dataJpg = camEx.PreviewImageToJpeg(data, 80)
iSt.InitializeFromBytesArray(dataJpg, 0, dataJpg.Length)
img.Initialize2(iSt)
img.GetPixel(img.Width/2, img.Height/2) ...
...

But as you know the preview frame is not high resolution. On an S4 is 1440x1080, that is not bad but very different from the 12Mpixel of the full frame picture. To do deeper analisys on the image I need the full frame picture, and to do that now I need to save and open it again (TOO SLOW).

 

[thedesolatesoul]

@JordiCP has done it here: http://www.b4x.com/android/forum/threads/cam-aka-studio-and-a-happy-new-year.36294/
I didnt think it could be done, but its been working incredibly well.

 

[Informatix]

The _Preview event is generated by the Camera, and what I have done in preview is to use the .PreviewImageToJpeg function to convert the Data() byte array

Yes, I saw that. My question is : why don't you reuse the same code for the data coming from the PictureTaken event? What result do you get?
To be more clear, replace these lines:
Dim dataJpg() As Byte, iSt As InputStream, img As Bitmap
dataJpg = camEx.PreviewImageToJpeg(data, 80)
by:
Dim iSt As InputStream, img As Bitmap
And replace dataJpg by data.

 

[mindcom]

why don't you reuse the same code for the data coming from the PictureTaken event? What result do you get?

Yes, already tried..
I receive an error, may be because of the image dimensions. PreviewImageToJpeg function seems to be specialized for preview frames.
Do you know who has written the Camera library? Is it possible to go deep on that code?

 

[mindcom]

@JordiCP has done it here: http://www.b4x.com/android/forum/threads/cam-aka-studio-and-a-happy-new-year.36294/

do you think JordiCP will share that code?

 

[thedesolatesoul]

do you think JordiCP will share that code?

You should read the first post again.
Maybe if you ask him nicely

 

[JordiCP]

Hi, I have just read this.

I will share a small library which can convert from NV21 to RGB888 in short (provided that you don't plan to do the same app, and rate mine with many stars ). Up to now I have been a "silent listener" here in the forum, learning from posts here and there from people as thedesolatesoul, Informatix and others, now it is time to contribute also.

But first a small disclaimer

I think it will be suitable for "pseudo-real-time", but take into account that overall performance will depend on resolution, device type and many other factors. There are also graphics libraries which will give you far better performance, but have never used them. The only reason is that I wanted to try doing it by myself as a part of my learning curve.

Hopefully will release it along this week, but can't promise anything.