让IjkPlayer支持插入自定义的GPU滤镜方法
最近因为工作的原因,需要提供一个将我们的AiyaEffectsSDK插入到IjkPlayer中的示例,就不得不好好看了下IjkPlayer的代码。在IjkPlayer中并没有提供设置自定义GPU滤镜的接口,所以最后只能自己动手,以求丰衣足食了。不得不说,Bilibili开源的这个IjkPlayer播放器的确非常强大,代码设计的非常清晰,仔细看看,能学到不少东西。
IjkPlayer源码获取及编译方法
源码地址,编译参考readme即可:
# 获取ijk源码 git clone https://github.com/Bilibili/ijkplayer.git ijkplayer-android # 进入源码目录 cd ijkplayer-android # checkout 最新版本 git checkout -B latest k0.8.0 # 执行脚本,此脚本会下载ijk依赖的源码,比如ffmpeg ./init-android.sh # 编译ffmpeg, all可以换成指定版本,如armv7a cd android/contrib ./compile-ffmpeg.sh clean ./compile-ffmpeg.sh all # 编译ijkplayer,all可以换成指定版本,如armv7a cd .. ./compile-ijk.sh all
IjkPlayer分析及修改
Android 版的IjkPlayer示例工程中,播放视频界面为tv.danmaku.ijk.media.example.activities.VideoActivity,在VideoActivity中使用的是IjkVideoView来播放视频的,位于tv.danmaku.ijk.media.example.widget.media包下。
IjkVideoView使用与Android的VideoView基本一致,在IjkVideoView中,设置视频源调用setVideoURI方法,而此方法又会调用private属性的openVideo方法。在openVideo方法中,会根据mSettings.getPlayer()的值创建一个IMediaPlayer:
public IMediaPlayer createPlayer(int playerType) { IMediaPlayer mediaPlayer = null; switch (playerType) { case Settings.PV_PLAYER__IjkExoMediaPlayer: { IjkExoMediaPlayer IjkExoMediaPlayer = new IjkExoMediaPlayer(mAppContext); mediaPlayer = IjkExoMediaPlayer; } break; case Settings.PV_PLAYER__AndroidMediaPlayer: { AndroidMediaPlayer androidMediaPlayer = new AndroidMediaPlayer(); mediaPlayer = androidMediaPlayer; } break; case Settings.PV_PLAYER__IjkMediaPlayer: default: { IjkMediaPlayer ijkMediaPlayer = null; if (mUri != null) { ijkMediaPlayer = new IjkMediaPlayer(); ijkMediaPlayer.native_setLogLevel(IjkMediaPlayer.IJK_LOG_DEBUG); if (mSettings.getUsingMediaCodec()) { ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "mediacodec", 1); if (mSettings.getUsingMediaCodecAutoRotate()) { ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "mediacodec-auto-rotate", 1); } else { ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "mediacodec-auto-rotate", 0); } if (mSettings.getMediaCodecHandleResolutionChange()) { ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "mediacodec-handle-resolution-change", 1); } else { ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "mediacodec-handle-resolution-change", 0); } } else { ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "mediacodec", 0); } //省略其他参数设置的的代码 } mediaPlayer = ijkMediaPlayer; } break; } if (mSettings.getEnableDetachedSurfaceTextureView()) { mediaPlayer = new TextureMediaPlayer(mediaPlayer); } return mediaPlayer; }
从上面代码中可以看到,在IjkVideoView中多处用到了mSettings,mSettings的值主要是用户设置的,通过SharedPreferences保存的,包括音视频解码设置、是否使用OpenSLES、渲染View等等。可以参看SettingsActivity界面。
根据playerType创建IjkMediaPlayer,前两类分别为google的ExoPlayer和Android的MediaPlayer。除此之外才是真正的创建的IjkPlayer。
mSettings中的其他参数最终会转换后通过IjkMediaPlayer的setOption方法进行设置,而IjkMediaPlayer.setOption又是直接调用native方法。进入IjkMediaPlayer可以发现,IjkMediaPlayer中的许多方法都是native方法,或者调用了native方法。
增加setGLFilter接口
在ijkmedia文件夹下全局搜索其中一个方法_setDataSource,得到内容大致如下:
F:\cres\C\ijkplayer-android\ijkmedia\ijkplayer\android\ijkplayer_jni.c: static void IjkMediaPlayer_setDataSourceAndHeaders( JNIEnv *env, jobject thiz, jstring path, jobjectArray keys, jobjectArray values) ... static void IjkMediaPlayer_setDataSourceFd(JNIEnv *env, jobject thiz, jint fd) { MPTRACE("%s\n", __func__); ... static void IjkMediaPlayer_setDataSourceCallback(JNIEnv *env, jobject thiz, jobject callback) { MPTRACE("%s\n", __func__); ... static JNINativeMethod g_methods[] = { { "_setDataSource", "(Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;)V", (void *) IjkMediaPlayer_setDataSourceAndHeaders }, { "_setDataSourceFd", "(I)V", (void *) IjkMediaPlayer_setDataSourceFd }, { "_setDataSource", "(Ltv/danmaku/ijk/media/player/misc/IMediaDataSource;)V", (void *)IjkMediaPlayer_setDataSourceCallback }, { "_setAndroidIOCallback", "(Ltv/danmaku/ijk/media/player/misc/IAndroidIO;)V", (void *)IjkMediaPlayer_setAndroidIOCallback },
即知,在Android版本的ijkplayer,入口即为ijkmedia\ijkplayer\android\ijkplayer_jni.c
在IjkMediaPlayer.java及ijkplayer_jni.c文件中增加setGLFilter方法:
//增加setGLFilter方法 static void IjkMediaPlayer_native_setGLFilter(JNIEnv *env, jclass clazz,jobject filter) { } // ---------------------------------------------------------------------------- static JNINativeMethod g_methods[] = { { "_setDataSource", "(Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;)V", (void *) IjkMediaPlayer_setDataSourceAndHeaders }, { "_setDataSourceFd", "(I)V", (void *) IjkMediaPlayer_setDataSourceFd }, { "_setDataSource", "(Ltv/danmaku/ijk/media/player/misc/IMediaDataSource;)V", (void *)IjkMediaPlayer_setDataSourceCallback }, //···省略其他方法 { "_setGLFilter", "(Ltv/danmaku/ijk/media/player/IjkFilter;)V", (void *) IjkMediaPlayer_native_setGLFilter }, };
渲染时回调用户设置的GLFilter相关方法
在ijkmedia/ijksdl/gles2/internal.h文件中,IJK_GLES2_Renderer结构体内增加:
typedef struct IJK_GLES2_Renderer { //... GLuint frame_buffers[1]; GLuint frame_textures[1]; int hasFilter=0; void (* func_onCreate)(void); void (* func_onSizeChanged)(int width,int height); void (* func_onDrawFrame)(int textureId); //... } IJK_GLES2_Renderer;
这三个方法即为IjkFlter的三个方法,将会在Jni里面,将Java中设置的IjkFilter对象的三个方法与之对应起来。
全局搜索glDraw搜索到只有一个在renderer.c的IJK_GLES2_Renderer_renderOverlay方法中,渲染工作也是此方法执行的。当然,IjkPlayer利用OpenGLES渲染时,会根据从视频中解码出来的数据具体格式来进行渲染,比如yuv420p、yuv420sp、rbg565等等诸多格式。具体在ijkmedia/ijksdl/gles2/下找到,renderer_rgb.c\renderer.yuv420p.c等等都是。
当用户在Java层设置了GLFilter时,GLFilter的三个方法应该在合适的时候被C回调,从名字可以看出来,这三个方法,和GLSurfaceView.Renderer接口中定义的三个方法其实是一样的。
具体在IJK_GLES2_Renderer_renderOverlay方法中的修改如下:
GLboolean IJK_GLES2_Renderer_renderOverlay(IJK_GLES2_Renderer *renderer, SDL_VoutOverlay *overlay) { /*用户设置了fitler,而且没有创建过framebuffer,创建framebuffer,依旧利用 IjkPlayer里面原来的流程,就yuv或rgb的数据,渲染到一个texture上面去,然后将 这个texture作为原始数据传递给java层进行处理。 */ if(renderer->hasFilter&&!renderer->frame_buffers[0]&&renderer->frame_width>0&&renderer->frame_height>0){ //创建一个texture,用来接受将不同格式的视频帧数据,渲染成一个纹理 glGenTextures(1,renderer->frame_textures); glBindTexture(GL_TEXTURE_2D,renderer->frame_textures[0]); glTexImage2D(GL_TEXTURE_2D,0,GL_RGBA,renderer->frame_width,renderer->frame_height,0,GL_RGBA,GL_UNSIGNED_BYTE,NULL); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE); glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE); glBindTexture(GL_TEXTURE_2D,0); //创建framebuffer来挂载纹理,以渲染视频帧 glGenFramebuffers(1,renderer->frame_buffers); glBindFramebuffer(GL_FRAMEBUFFER,renderer->frame_buffers[0]); glFramebufferTexture2D(GL_FRAMEBUFFER,GL_COLOR_ATTACHMENT0,GL_TEXTURE_2D,renderer->frame_textures[0],0); // int r; // if ((r = glCheckFramebufferStatus(GL_FRAMEBUFFER)) != GL_FRAMEBUFFER_COMPLETE) // { // ALOGE("wuwang: Error in Framebuffer 0x%x", r); // }else{ // ALOGE("wuwang: glCheckFramebufferStatus 0x%x", r); // } glBindFramebuffer(GL_FRAMEBUFFER,0); //调用Java传递进来的Filter的onCreated方法及onSizeChanged方法 renderer->func_onCreated(); renderer->func_onSizeChanged(renderer->frame_width,renderer->frame_height); ALOGE("wuwang: create frame_buffers and textures %d,%d",renderer->frame_width,renderer->frame_height); } //用户设置了Filter,就挂载frameBuffer,否则就按照原来的流程直接渲染到屏幕上 if(renderer->hasFilter&&renderer->frame_buffers[0]){ GLint bindFrame; glGetIntegerv(GL_FRAMEBUFFER_BINDING,&bindFrame); ALOGE("wuwang: default frame binding %d",bindFrame); glBindFramebuffer(GL_FRAMEBUFFER,renderer->frame_buffers[0]); // glFramebufferTexture2D(GL_FRAMEBUFFER,GL_COLOR_ATTACHMENT0,GL_TEXTURE_2D,renderer->frame_textures[0],0); /* 这句一定要加上,否则无法增加了Filter之后,启用了其他GLProgram,无法渲染原始视频到Texture上去了 */ IJK_GLES2_Renderer_use(renderer); } if (!renderer || !renderer->func_uploadTexture) return GL_FALSE; glClear(GL_COLOR_BUFFER_BIT); IJK_GLES2_checkError_TRACE("glClear"); ALOGE("wuwang: frame buffer id:%d",renderer->frame_buffers[0]); GLsizei visible_width = renderer->frame_width; GLsizei visible_height = renderer->frame_height; if (overlay) { visible_width = overlay->w; visible_height = overlay->h; if (renderer->frame_width != visible_width || renderer->frame_height != visible_height || renderer->frame_sar_num != overlay->sar_num || renderer->frame_sar_den != overlay->sar_den) { renderer->frame_width = visible_width; renderer->frame_height = visible_height; renderer->frame_sar_num = overlay->sar_num; renderer->frame_sar_den = overlay->sar_den; renderer->vertices_changed = 1; } renderer->last_buffer_width = renderer->func_getBufferWidth(renderer, overlay); if (!renderer->func_uploadTexture(renderer, overlay)){ return GL_FALSE; } } else { // NULL overlay means force reload vertice renderer->vertices_changed = 1; } GLsizei buffer_width = renderer->last_buffer_width; if (renderer->vertices_changed || (buffer_width > 0 && buffer_width > visible_width && buffer_width != renderer->buffer_width && visible_width != renderer->visible_width)){ if(renderer->hasFilter&&renderer->frame_buffers[0]){ renderer->func_onSizeChanged(renderer->frame_width,renderer->frame_height); } renderer->vertices_changed = 0; IJK_GLES2_Renderer_Vertices_apply(renderer); IJK_GLES2_Renderer_Vertices_reset(renderer); IJK_GLES2_Renderer_Vertices_reloadVertex(renderer); renderer->buffer_width = buffer_width; renderer->visible_width = visible_width; GLsizei padding_pixels = buffer_width - visible_width; GLfloat padding_normalized = ((GLfloat)padding_pixels) / buffer_width; IJK_GLES2_Renderer_TexCoords_reset(renderer); IJK_GLES2_Renderer_TexCoords_cropRight(renderer, padding_normalized); IJK_GLES2_Renderer_TexCoords_reloadVertex(renderer); } glDrawArrays(GL_TRIANGLE_STRIP, 0, 4); IJK_GLES2_checkError_TRACE("glDrawArrays"); //用户设置了Filter,就取消挂载FrameBuffer,并调用Filter的onDrawFrame方法。 if(renderer->hasFilter&&renderer->frame_buffers[0]){ glBindFramebuffer(GL_FRAMEBUFFER,0); renderer->func_onDrawFrame(renderer->frame_textures[0]); // renderer->func_onDrawFrame(renderer->plane_textures[0]); } return GL_TRUE; }
GLFilter从设置到调用实现分析
上面已经完成的接口的编写,也做好执行的编写。现在需要将接口传递进来的GLFilter的三个方法与执行的三个方法对应起来,才能是用户的Filter真正发挥作用。
IJK_GLES2_Renderer的创建是根据SDL_VoutOverlay来的,在查找SDL_VoutOverlay从哪里来的,一路可以搜索到ijkmedia/ijksdl/android/ijksdl_vout_android_nativewindow.c中的func_create_overlay方法:
static SDL_VoutOverlay *func_create_overlay(int width, int height, int frame_format, SDL_Vout *vout) { SDL_LockMutex(vout->mutex); SDL_VoutOverlay *overlay = func_create_overlay_l(width, height, frame_format, vout); SDL_UnlockMutex(vout->mutex); return overlay; }
SDL_VoutOverlay的创建和SDL_Vout有关,再查找SDL_Vout的来源,可以找到
ijkmedia/ijksdl/android/ijksdl_vout_android_nativewindow.c中的SDL_VoutAndroid_CreateForANativeWindow:
SDL_Vout *SDL_VoutAndroid_CreateForANativeWindow() { SDL_Vout *vout = SDL_Vout_CreateInternal(sizeof(SDL_Vout_Opaque)); if (!vout) return NULL; SDL_Vout_Opaque *opaque = vout->opaque; opaque->native_window = NULL; if (ISDL_Array__init(&opaque->overlay_manager, 32)) goto fail; if (ISDL_Array__init(&opaque->overlay_pool, 32)) goto fail; opaque->egl = IJK_EGL_create(); if (!opaque->egl) goto fail; vout->opaque_class = &g_nativewindow_class; vout->create_overlay = func_create_overlay; vout->free_l = func_free_l; vout->display_overlay = func_display_overlay; return vout; fail: func_free_l(vout); return NULL; }
它的初始化,再没啥可以关联了,那就只能找调用它的地方了。搜索后发现SDL_VoutAndroid_CreateForANativeWindow只在ijkmedia\ijksdl\android\ijksdl_vout_android_surface.c的SDL_VoutAndroid_CreateForAndroidSurface方法:
SDL_Vout *SDL_VoutAndroid_CreateForAndroidSurface() { return SDL_VoutAndroid_CreateForANativeWindow(); }
看来是被包了一层皮,那就接着查找SDL_VoutAndroid_CreateForAndroidSurface,搜索到调用它的为ijkmedia\ijkplayer\android\ijkplayer_android.c的ijkmp_android_create方法:
IjkMediaPlayer *ijkmp_android_create(int(*msg_loop)(void*)) { IjkMediaPlayer *mp = ijkmp_create(msg_loop); if (!mp) goto fail; mp->ffplayer->vout = SDL_VoutAndroid_CreateForAndroidSurface(); if (!mp->ffplayer->vout) goto fail; mp->ffplayer->pipeline = ffpipeline_create_from_android(mp->ffplayer); if (!mp->ffplayer->pipeline) goto fail; ffpipeline_set_vout(mp->ffplayer->pipeline, mp->ffplayer->vout); return mp; fail: ijkmp_dec_ref_p(&mp); return NULL; }
可以看到ijkmp_android_create返回了一个IjkMediaPlayer,这个在Java中也有一个这样的类,那么曙光应该就不远了。
再搜索ijkmp_android_create,结果调用它的只有ijkmedia\ijkplayer\android\ijkplayer_jni.c中的IjkMediaPlayer_native_setup方法,到了这里,就可以将IjkFilter传递下去了。
在上面的过程中,可以看到从renderer.c到jni,IJK_GLES2_Renderer的创建,依赖SDL_VoutOverlay。SDL_VoutOverlay的创建依赖SDL_Vout。SDL_Vout是FFPlayer的成员,而FFPlayer又是IjkMediaPlayer的成员变量。Java层的IjkMediaPlayer依赖于IjkMediaPlayer
从GLFilter到IJK_GLES2_Renderer
根据上面的分析,可以知道,在jni中增加的setGLFilter的方法中,我们可以将GLFilter的方法传递给IjkMediaPlayer->FFPlayer->SDLVout,然后再传递给SDL_VoutOverlay,再由SDL_VoutOverlay传递给IJK_GLES2_Renderer即可。这样将增加Filter的功能增加进去了,也不会影响IjkPlayer的流程,让IOS同样能够快速的实现增加GPU滤镜的功能。
先在SDL_VoutOverlay和SDL_Vout中的结构体定义中(ijkmedia/ijksdl/ijksdl_vout.h文件中)同样加入在IJK_GLES2_Renderer中增加的成员:
struct SDL_VoutOverlay { //... int hasFilter; void (* func_onCreated)(void); void (* func_onSizeChanged)(int width,int height); int (* func_onDrawFrame)(int textureId); //... }; struct SDL_Vout { //... int hasFilter; void (* func_onCreated)(void); void (* func_onSizeChanged)(int width,int height); int (* func_onDrawFrame)(int textureId); //... };
然后在上述几个方法,分别完成这几个成员数据从SDL_Vout到SDL_VoutOverlay,再到IJK_GLES2_Renderer的传递。
分别为:
ijkmedia\ijksdl\gles2\renderer.c文件中IJK_GLES2_Renderer_create方法
IJK_GLES2_Renderer *IJK_GLES2_Renderer_create(SDL_VoutOverlay *overlay) { if (!overlay) return NULL; //中间省略... renderer->format = overlay->format; //增加的内容 renderer->hasFilter=overlay->hasFilter; renderer->func_onCreated=overlay->func_onCreated; renderer->func_onSizeChanged=overlay->func_onSizeChanged; renderer->func_onDrawFrame=overlay->func_onDrawFrame; return renderer; }
ijksdl\android\ijksdl_vout_android_nativewindow.c文件中func_create_overlay方法
static SDL_VoutOverlay *func_create_overlay(int width, int height, int frame_format, SDL_Vout *vout) { SDL_LockMutex(vout->mutex); SDL_VoutOverlay *overlay = func_create_overlay_l(width, height, frame_format, vout); //增加的内容 overlay->hasFilter=vout->hasFilter; overlay->func_onCreated=vout->func_onCreated; overlay->func_onSizeChanged=vout->func_onSizeChanged; overlay->func_onDrawFrame=vout->func_onDrawFrame; SDL_UnlockMutex(vout->mutex); return overlay; }
最后还需要完成SDL_Vout中这几个成员的赋值,并调用Java传入的GLFilter对象的相关方法(ijkplayer_jni.c文件中):
static JNIEnv * mEnv; static jobject mFilter; static jmethodID onCreatedMethod; static jmethodID onSizeChangedMethod; static jmethodID onDrawFrameMethod; void onCreated(){ if(!mEnv){ (*g_jvm)->AttachCurrentThread(g_jvm,&mEnv,NULL); jclass filterClass=(*mEnv)->GetObjectClass(mEnv,mFilter); onCreatedMethod=(*mEnv)->GetMethodID(mEnv,filterClass,"onCreated","()V"); onSizeChangedMethod=(*mEnv)->GetMethodID(mEnv,filterClass,"onSizeChanged","(II)V"); onDrawFrameMethod=(*mEnv)->GetMethodID(mEnv,filterClass,"onDrawFrame","(I)I"); (*g_jvm)->DetachCurrentThread(g_jvm); } if(onCreatedMethod){ (*g_jvm)->AttachCurrentThread(g_jvm,&mEnv,NULL); (*mEnv)->CallVoidMethod(mEnv,mFilter,onCreatedMethod); (*g_jvm)->DetachCurrentThread(g_jvm); } } void onSizeChanged(int width,int height){ if(onSizeChangedMethod){ (*g_jvm)->AttachCurrentThread(g_jvm,&mEnv,NULL); (*mEnv)->CallVoidMethod(mEnv,mFilter,onSizeChangedMethod,width,height); (*g_jvm)->DetachCurrentThread(g_jvm); } } int onDrawFrame(int textureId){ if(onDrawFrameMethod){ (*g_jvm)->AttachCurrentThread(g_jvm,&mEnv,NULL); int ret=(*mEnv)->CallIntMethod(mEnv,mFilter,onDrawFrameMethod,textureId); (*g_jvm)->DetachCurrentThread(g_jvm); return ret; } return textureId; } /*注意不能直接保存env,filter然后在onDrawFrame等方法中使用,因为这三个方法的调用与setGLFilter不是在同一个线程中*/ static void IjkMediaPlayer_native_setGLFilter(JNIEnv *env, jobject clazz, jobject filter) { if(mFilter){ (*env)->DeleteGlobalRef(env,mFilter); } IjkMediaPlayer *mp = jni_get_media_player(env, clazz); if(filter!=NULL){ mFilter=(*env)->NewGlobalRef(env,filter); mp->ffplayer->vout->hasFilter=1; mp->ffplayer->vout->func_onCreated=onCreated; mp->ffplayer->vout->func_onSizeChanged=onSizeChanged; mp->ffplayer->vout->func_onDrawFrame=onDrawFrame; }else{ mp->ffplayer->vout->hasFilter=0; } }
至此,Java一直到sdl中的renderer就算连通了。在Java中的处理就和GLSurfaceView设置Renderer基本一致了,不同的是,我们给IjkPlayer中增加的GLFilter,已经提供了一个原始的视频图像作为onDrawFrame的参数,在GLFilter中,只需要处理这个Texture并渲染出来就可以了。
插入滤镜示例
将修改后的代码重新编译下,编译后的库会自动更新到Ijkplayer的Android工程下,设置自定义的滤镜后,不出意外就可以看到效果了。以下分别为原始视频、黑白滤镜处理后的视频、增加了AiyaEffectsSDK并设置了特效的视频效果,因为CSDN对图片大小限制的问题,都是截取了一小段:
工程太大了,修改的地方也不算多,就不上传代码了。有需要的朋友根据以上流程下载ijkplayer源码自行修改即可,同时也可以看看Ijkplayer的源码。
以上这篇让IjkPlayer支持插入自定义的GPU滤镜方法就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持我们。