CTP project and CISC4900 Presentation, Signalyze

hi Welcome to our presentation for 4900 today our topic was a shaliz it's a

program it's a project we did throughout semester our teammates are Marco Chen

Regal B pun and me so our project overview is uh Liberto

doing ta spe and my uh teammates Marco Chen doing the gestra hand handun Mark

detection sections so here is our uh recapturing of our uh presentation

nights and this is [Music]

us all right now let's start with the

presentation so we first go through planning [Music]

Toto work on the TA of speech documentation and recommendation for project huging face and uh stream

streamate testing for test text to speech and project suggestions the marro

worked on open CV sign language training data collection local testing and implementation on streamlet I'm working

on smaller project such as uh H5 training to testing different platforms

for usability and put the project together integrating text to speech open

CV and also working on exploring liosa for future a

augmentation so today's presentation format uh going through like this first I will introducing a voice processing uh

testing so this is the most important part of my learning where I learned all the stuffs from this uh video processing

for music and also for How to Train H5 file a train different datas and the

second stage is like I going testing liosa and it's uses on music analysis

which will lead to my understanding of a voice processing which later on uh help

me integrating this text of speech and the uh for the integration of the sign

language and text of speech and to understanding basically how this a of speech things and the audio handling

work so the third step was introducing image processing and then video processing and then talk about platforms

and failures and the last last steps we will present our project and also a

recapturing of our presentation night with my teammates so now let's introduce my work

and for the project so for my part my contribution was a planning so uh rerto

and M have formatting uh documentary to listing what do we need to use for this

project and I'm going to plan and and making smaller project to make each step work and then we put the things together

similar like how do you build this uh like puzzle things so the first thing is

understanding basic steps and once that we understanding the basic concepts we can finding different resources and the

coding and testing and then the testing platform including Unity flask hugging

face Streamlight a uh Amazon Cloud and also render heru and also uh various

different platform per for implementation experimenting on liosa and currently in progress and these

things uh make it work tonight actually it's the small Personal

Achievement so the problem statement the sign language is a crucial communication

tools for the people with death and hard hearing communities but many people are

not fluent in it so brezing this Gap through technology can enhance this incl

inclusivity and accessibility so that's why we come up with this uh idea about

using open CV to recognizing hand system so our solution is first we

process a system that can capture shine gestures and using a camera and then recognize gestures through machine

learning models and then convert this gesture to different text and text to spe in real

time so the technology we use is open CV which was like a real time video

processing uh app and also media pipe was really helpful like introduced by

ham and also it's hand tracking and gesture recommendation system penser flow and kalas is uh handling deep

learning and model development and finally we're using python libraries to

uh using this txos spe and NP and also flask so my first step starting with

learning something small so handling WS is the most important part because like we have to uh go through one dimensional

to three dimensional know so the voice is the basic part and then the image using this uh CNN formats and different

formats and then after uh voice and the image we go through video and integrating how video and the voice can

integrate together so for me music study is part of my speciality like in this

small project so I using music study and voice handling inspired by Ham's

repository for emotional Rec recognization system to start this project first let's go through uh Ham's

uh really amazing report story so husam is our teaching assistant which has been

really helpful and he's so great so they are doing something using machine

learning and librosa to recognize different speech and generate General uh generalizing emotions so using this

emotion to detect uh other voices using this machine learning model and what I

learned from this uh really amazing model mods like they using inspect and explore data and selecting

engineering features build and train models evaluate models so basically they uh import

different libraries and using machine learning to learn things and they have this P PBE to presenting audio segment

audio that can be manipulated using using python code the first step is inspect explore

data so uh they they training with really great data like from K but for me

was like I want to learn something about my own music so I doing my own style of training I will show you later so

importing data size is another uh issues because like they always have formatting issues things and preparing data and uh

revness redness so this this part was basically dealing with uh data labeling

data I think data cleaning and data like preparation but for my part I don't

really need those things because I label my own uh data I know was my emotion so I put into different folder and then

just check later and then preparing data for creating this data sit this is another

stat and after you labeling different things for example female male and neutral emotion and all those uh things

then you replace read uh like replace different kind of like type of things

for example replace female with female and these type of things but for my case I doing with small data and my own data

so it doesn't apply so their project is more ambitious but our project is a little bit smaller so yeah then they

using V uh data visualization currently I don't have this uh feature yet but I

my my my my own data was only dealing with six different prototype I will show you

later yeah so basically they are using liosa domestic uh do domestic Learning

System to mapping this different type of uh waves and then you you learning this

different type of things for example H and also like the the seconds so they can study these

type of things uh like to recognize a different partn they haven't seen before so use these things to predict things

and this is really great system and for this this spectrum things I already seen

the things during Professor cohan's class with electronic music L master so

this is also another very interesting thing in learning different sound

Yeah so basically this is the idea and this is from Ham's temp Ham's repo I learning extensively and now let's going

back to this learning process so the learning uh FF ffmpeg is another uh

formatting thing so this formatting can convert in between MP3 to waveform because waveform is bigger not bigger

like more specific MP3 is smaller and then I also learning so many things with

profess Cohen on this Timber things and how how Tempo things and how this uh electronic things going through this uh

other things I forgot what's the software was the software that you can paste uh track there and they reflecting

different like uh pictures and also wait sorry labeling my

own data is another process I label my own data I'm going to show you my basic data labeling things going on

so for my small project I have this uh music data so all this music I pick up

different music I compos during the past three years cuz I know those styles are unique to me so I label them as as a

different labels and then put this uh music sty in there to learning from so that's how I label my own data

and then supposed to be generating a CSV format to uh recording this data point

for example they're using different software extracting Temple and the chroma and also those like picture line

things and then up after this process we can generating a H5 machine learning

model and they using this model to predicting other things so now let's going back to topic

so what's is ffmp eg format ffmpeg is a

powerful open source tool for handling mtim media data it can convert audio and

video formats and then resample re-encode and normalize this audio file so basically for my case it's

like converting between MP3 and wave because like the wave s format was better with a better quality and it's

easier to extracting this uh music information so for handling unusual

audios yeah there is like when we dealing with the data learning sorry

when we dealing with this uh machine learning and data labeling there will always be this uh conflicting things and

they were resulting the fail is not reading into C CSV or don't recording

correctly so for me was PL to false read all fils and there's a process I learned

called fallback mechanism it's like if a Fisher have this Temple mfcc and chromas

basically they are different wave recording different wave things if it it fails and has unexpected Dimensions this

Dimensions things that uh in my best knowledge is like array so for example if Temple only have one array and then

the other format have two array so this things will not match so I have to force into some data into it so for my

solution is like for any problematic fil inro output has consistent Dimensions

this is the first issues and if U something for example like electronic music or from some of my music have this

really weird tones they will not be recognized then I put it as zeros on alternative placeholder so this zero is

still valid because um when they read through this uh datas they will still recognize a part part and use it use it

to predicting other partn and no no skipping so sometimes they skip these fils when it's not working so we have to

write the code saying this loog issues must be like a continue processing every

fil to make sure every fil have recorded something even if something was not showing there they put a zero there

instead of uh like skip it skip this whole fil to mess up this whole

fils and the next stage is called debugging the debug debugging fails usually they have this uh uniform shape

for all features so it's basically they have Temple mfcc chroma is reshaped to

fix this dimension for example Tempo has one row and mfcc has 13 rows and chroma

has 12 rows so I will skip this uh process this is another like kind of like personal discussion so this is my

backlog so the biggest issue I might was this uh like something called dimension thing because for example these things

when they uh when they produce reading my uh one of the piece Professor zerio

like reference or experimental things they don't have the same dimensions but array in the index zero has two

dimensions and the other has one so I have to that's why I went through this process to f everything through and

eventually this thing more rate for example they say Temple shape is one 1

5,000 and the mfcc shape is 13 5,000 the chroma shape is this like all of these

things will give you a number and this number is supposed to convert into CSV

but CSV failed I'm going to explain why because there are fail types the npy is a binary fail for my specific to NP and

also optimized for strong arrays for metadata for example this strong arrays and this is why I'm going to skip this

uh thing you can read [Music] later so for analyzing why my CSV file

didn't work so why it didn't work because our features have multi- dimensionals for example 26 5,000 which

is a CS can't handle this uh things like by itself CU CSV was like one dimension

so for flattering those arrays into a CVS would lose the structure integrating in this data and complicate processing

and model training so that's why we are using NP NP structure inside of this

structure and Mark also come up with other different structures which is really great for his final hand

recognization system from media pipe so now let's go to next micro stpe

now we produce our first uh npy sorry npy format this suppos this just imagine

this is similar as CSV so they listing all the stas like which is this this

this form what is this point what what number is it and we can now train H5 to

recognize those parns now we plan for training the

models the training in the models first were load and uh the data site load py F

and separated features and label them and then we train them split into training and testing side and after

these things we'll be able to produce H5 file so how it works basically the motor

learns pattern from these input features and those features capture ke characteristic my music style such as

Rhythm tone and Timber and all those are comp compiled from this chroma uh

packages so this is my six different I call it traditional centin and crab

class Canon Innovative existence BL and the reference so experimental so each

have the uh different ideas they doesn't exist in any other sty so after learning

my own partner they will produce six stells labeled at 0 to

5 now I make it into an h55 fil so now we're going through this training

process using neural network map mapping to capture this uh part

[Music] so now we have this H5 file using my uh

CSV which was npy file like the two dimensional things and then they will give you a H5 file this H5 f is the most

important part of any modor training because like usually other people train from tens of thousands files into this

H5 file so people can borrow this model to train their own model and this is uh

I learning how to do these things from scratch because I really like uh Ham's

idea like back then we don't have huging face we don't have all that we have to stream by ourself this is why uh I

really like this uh things so I start to train start to train things by myself so

not every h55 will end up in success so there was only two working one is this

the other one was the handshake I'll show you really quick now let's testing these things on streamlets

so here is my project let's show let me show you [Music]

see now they will show you this six sty right and then you drag a random video

with a either MP3 or wave and they will analyze a wh of this six sty integrating

with each other for example let me put my own music there

put something run regular

see now it's analyzing based on my H5 [Music]

file and what it will produce it predicting like we have this uh 7.4% of

this sty they were using this part to recognize which six St integrating with

other so they recognize my uh this piece was most close to this uhx there which

was 32 so this this is how how it end like ended so inov way of existence 36%

so they were learning how different music style like related correlated to each other based on this pattern and

this is my Personal Achievement on this small things sorry I forgot how

to yeah let me stop here

now let's back to the main project section two so the section two is image processing and media pipe for video

processing exploring media pipe and hand tracking and Analysis uh different like

data using Training Method the method we use is data preparation data training and real time

integration so the fail what fails is B the unity I will not show it here

because this on file is even more complicated this H5 H5 only have two success by myself usually people borrow

from other library but like other people's Library could be bed and you myself you okay so like my teammate H

really well but so for now I'm just going to talking about like U my contribution on this small micro project

so the main step in process is cleaning data and uses this is my friends doing my teammates doing this type of things

like did handle it really well and then Tex of speech uh do it

really well we have different formatting and I'm using uh Max native so now we

have backup plan backup my backup plan was to using Unity developing this like a number recognization system so we we

start from basic number to hand to video to V uh to audio and integrating all of

them together so now what does our first prototype do our first prototype is

input we capture using this media pipe for this finger points and then we learn

these finger point patterns so what's the potential from this basic prototype it can of course

learning a sign language helping people helping people with disability and also for music this sign language can help us

learning conduct comparisons and also for gaming we can using this gaming hand controller and for new music art project

some people already come up with these things and also can anise just play for fun like I develop game we'll show you

later so H5 versus Onix they are two different formats one was H5 one was onx

so the H5 hierarchy data format was using this tensorflow caras and onx was

like mostly for with unity bar things but it didn't work so I'm mostly using H5 in this

case the solutions and steps focus on cleaning data and testing real time

integration and refining the text spe and then you're using unity and finally you tou with flask flask is also Ham's

idea like ham is really great I so many great things we learned so many good things from

him the unity said that bakuda was something I failed I want to try more because Unity was better than other

platforms this is the bakuda testing and H5 what's is H5 as I show you before H5

is called hierarchy data format these things using the reading through different format for example c c CVS and

also the phy things like I show you before the two dimensional things I call

it like that user the use case is commonly used for tensor flow and kasas and also they can start different neural

networks ways training figures the onx format was better for uh

baracuda and also us AI P to sensor flow and different

platforms so how do we convert in between we creating a code converting between H5 and onx optimization this

things also good with GPU and mobile things there are also other

formats how do we converting different formats we converting from H5 to onx using this B no using this PP in style

these things and then we copy paste the code like this the code like this the T

onx export model input tens model onx and then we you we creating a converter

converting these things into the other format but these things always fail for me I don't know why but in the future

we'll continue exploring the flux was some best idea so I learning so much

with this flux I will show you really quick later as I show you I'm going to uh like

tell you how I set up this flask so basically flask a was like deploying a

flask a using different platforms so my solution was using uh rer and my

teammates also have this uh kind of like streamlets hugging face things now let's

Tes in different platforms so let me close one of them first

[Music]

yeah first let me show you about this uh

basic testing [Music] system first we test if this uh hand

shaking things working or not [Music]

wait wait wait wait this student supposed to be like this

[Music]

or I'm show you another one [Music]

sorry so for this project

yeah this project will show you this will have generating a different uh format and the guys they're playing a

guys in game with you for example this this hand this hand is the uh math and

this is the F and this is the math so they were playing something similar to

this uh Cesar rock with you and then they will generate randomly generate something so I also trained these things

one by one with uh using this media pipe to recognize recognize different hand marks and then I make 150 different

pictures to learning this things so this is my unique design on this similar to

this uh model par and recognization system now let's show sorry my computer is a little bit slow so these things

will be really uh unstable for example I'm using math you choose a mice you

win give a second will say mice is0 75 and then they will randomly generate

something see if you win or not you win now let's do hand you can't even move your hand like this you choose the fool

no I choose the fool computer wins give it a little bit time you choose the fool 100% this is easy to

this is just a p sign and now we have a cat let's see the cat was the hard one

you have curving finger no they choose you choose a cat computer wins see you choose a

cat it's a tie you choose a cat you win yeah this is basically what

happened I'm stop it just in case draining my computer and they were

generating here so now for the last part I would

like to show our final project at the presentation night and thanks thankfully

my teammates uh RTO and Marco CH and me was working on these things so for the

last I would like to show our recapturing for maros Shen's work so here is maros

[Music]

Shen sorry now let's uh introduce our project

you can say something so uh actually our projects are related about um the comp

computer about on the data science so basically we uh use the uh medium P

which is a uh uh open source library from uh uh Pyon uh which is relate to

the uh compation recognition so um it's providing a a model that uh we can use

our own data set to chain it to recognize different uh label So based on

those label we put in the JP picture or MPG picture in those label and then put

in the model to change it after we change we can also decide like how

precise we want and also we can well turn the model in the future to work

small side but for this project now we did is we change it we use the data set

uh the ASL side it a to c to recognize

all that hand so uh now I'm going to show a demo on

the here on my so I'm going to side the light camera F so basically

uh so uh oh that's perfect so uh we have a

side here you so first of all uh if put

our hand there's a landmark uh detect on the finger so it have like 21 phone

yeah card on your hand recognize is a hand shake yeah definitely you can do it

like like face shap something like that but now uh I can try to do a hand side to let the model weize my uh side

language so I do the a so it give uh the decided WID and also the how confid the

confidence how precise a model Rec this okay and then uh let's try two different

side now okay uh

[Music]

okay this is a s no this is s s like

this yeah that's e okay this a uhhuh this a this is a okay okay this is B if

you close the finger you close the finger that will

be okay and then uh if you make a finger cross that

[Music] be and you can try to be

and and then three finger is W okay that's great

the letter will only keep the record of the change cover

if your size keep the same it don't work so try to keep like less

M [Music] okay thank you that'll be all thank you

for uh watching our video hope you enjoy have a nice day