𝗛𝗼𝘄 𝘄𝗲 𝗺𝗮𝗱𝗲 𝗚𝗿𝗮𝗱𝗶𝗼 𝗳𝗮𝘀𝘁𝗲𝗿 𝗯𝘆... 𝘀𝗹𝗼𝘄𝗶𝗻𝗴 𝗶𝘁 𝗱𝗼𝘄𝗻!
text
𝗛𝗼𝘄 𝘄𝗲 𝗺𝗮𝗱𝗲 𝗚𝗿𝗮𝗱𝗶𝗼 𝗳𝗮𝘀𝘁𝗲𝗿 𝗯𝘆... 𝘀𝗹𝗼𝘄𝗶𝗻𝗴 𝗶𝘁 𝗱𝗼𝘄𝗻!
new_line
new_line
About a month ago, 
text
About a month ago, 
@oobabooga
mention
oobabooga
 (who built the popular text generation webui) reported an interesting issue to the Gradio team. After upgrading to Gradio 4, 
text
 (who built the popular text generation webui) reported an interesting issue to the Gradio team. After upgrading to Gradio 4, 
@oobabooga
mention
oobabooga
 noticed that chatbots that streamed very quickly had a lag before their text would show up in the Gradio app. 
text
 noticed that chatbots that streamed very quickly had a lag before their text would show up in the Gradio app. 
new_line
new_line
After some investigation, we determined that the Gradio frontend would receive the updates from the backend immediately, but the browser would lag before rendering the changes on the screen. The main difference between Gradio 3 and Gradio 4 was that we migrated the communication protocol between the backend and frontend from Websockets (WS) to Server-Side Events (SSE), but we couldn't figure out why this would affect the browser's ability to render the streaming updates it was receiving.
text
After some investigation, we determined that the Gradio frontend would receive the updates from the backend immediately, but the browser would lag before rendering the changes on the screen. The main difference between Gradio 3 and Gradio 4 was that we migrated the communication protocol between the backend and frontend from Websockets (WS) to Server-Side Events (SSE), but we couldn't figure out why this would affect the browser's ability to render the streaming updates it was receiving.
new_line
new_line
After diving deep into browsers events, 
text
After diving deep into browsers events, 
@aliabid94
mention
aliabid94
 and 
text
 and 
@pngwn
mention
pngwn
 made a realization: most browsers treat WS events (specifically the 
text
 made a realization: most browsers treat WS events (specifically the 
WebSocket.onmessage
`WebSocket.onmessage`
inline_code
 function) with a lower priority than SSE events (
text
 function) with a lower priority than SSE events (
EventSource.onmessage
`EventSource.onmessage`
inline_code
 function), which allowed the browser to repaint the window between WS messages. With SSE, the streaming updates would stack up in the browser's event stack and be prioritized over any browser repaint. The browser would eventually clear the stack but it would take some time to go through each update, which produced a lag.
text
 function), which allowed the browser to repaint the window between WS messages. With SSE, the streaming updates would stack up in the browser's event stack and be prioritized over any browser repaint. The browser would eventually clear the stack but it would take some time to go through each update, which produced a lag.
new_line
new_line
We debated different options, but the solution that we implemented was to introduce throttling: we slowed down how frequently we would push updates to the browser event stack to a maximum rate of 20/sec. Although this seemingly “slowed down” Gradio streaming, it actually would allow browsers to process updates in real-time and provide a much better experience to end users of Gradio apps. 
text
We debated different options, but the solution that we implemented was to introduce throttling: we slowed down how frequently we would push updates to the browser event stack to a maximum rate of 20/sec. Although this seemingly “slowed down” Gradio streaming, it actually would allow browsers to process updates in real-time and provide a much better experience to end users of Gradio apps. 
new_line
new_line
See the PR here: 
text
See the PR here: 
https://github.com/gradio-app/gradio/pull/7084
https://github.com/gradio-app/gradio/pull/7084
link
new_line
new_line
Kudos to 
text
Kudos to 
@aliabid94
mention
aliabid94
 and 
text
 and 
@pngwn
mention
pngwn
 for the fix, and to 
text
 for the fix, and to 
@oobabooga
mention
oobabooga
 and 
text
 and 
@pseudotensor
mention
pseudotensor
 for helping us test it out!
text
 for helping us test it out!
new_line
𝗛𝗼𝘄 𝘄𝗲 𝗺𝗮𝗱𝗲 𝗚𝗿𝗮𝗱𝗶𝗼 𝗳𝗮𝘀𝘁𝗲𝗿 𝗯𝘆... 𝘀𝗹𝗼𝘄𝗶𝗻𝗴 𝗶𝘁 𝗱𝗼𝘄𝗻!

About a month ago, @oobabooga (who built the popular text generation webui) reported an interesting issue to the Gradio team. After upgrading to Gradio 4, @oobabooga noticed that chatbots that streamed very quickly had a lag before their text would show up in the Gradio app. 

After some investigation, we determined that the Gradio frontend would receive the updates from the backend immediately, but the browser would lag before rendering the changes on the screen. The main difference between Gradio 3 and Gradio 4 was that we migrated the communication protocol between the backend and frontend from Websockets (WS) to Server-Side Events (SSE), but we couldn't figure out why this would affect the browser's ability to render the streaming updates it was receiving.

After diving deep into browsers events, @aliabid94 and @pngwn made a realization: most browsers treat WS events (specifically the `WebSocket.onmessage` function) with a lower priority than SSE events (`EventSource.onmessage` function), which allowed the browser to repaint the window between WS messages. With SSE, the streaming updates would stack up in the browser's event stack and be prioritized over any browser repaint. The browser would eventually clear the stack but it would take some time to go through each update, which produced a lag.

We debated different options, but the solution that we implemented was to introduce throttling: we slowed down how frequently we would push updates to the browser event stack to a maximum rate of 20/sec. Although this seemingly “slowed down” Gradio streaming, it actually would allow browsers to process updates in real-time and provide a much better experience to end users of Gradio apps. 

See the PR here: https://github.com/gradio-app/gradio/pull/7084

Kudos to @aliabid94 and @pngwn for the fix, and to @oobabooga and @pseudotensor for helping us test it out!

👍
kramp osanseviero pngwn Ifeanyi xpgx1 BramVanroy DrHouseFan-315 samusenps clem phanes gate369 fffiloni sbarman25 taufiqdp akhaliq ysharma davanstrien gsarti adamm-hf julien-c merve hysts freddyaboulton
❤️
clem akhaliq ysharma kramp santiviquez gsarti adamm-hf julien-c
🤗
DrHouseFan-315 clem akhaliq ysharma gsarti adamm-hf julien-c
🤯
adamm-hf julien-c sbrandeis
🤝
julien-c
61868ce808aae0b5499a2a95
https://cdn-avatars.huggingface.co/v1/production/uploads/61868ce808aae0b5499a2a95/F6BA0anbsoY_Z7M1JrwOe.jpeg
Sylvain Filoni
fffiloni
user
60d2dc1007da9c17c72708f8
https://cdn-avatars.huggingface.co/v1/production/uploads/1624431552569-noauth.jpeg
yuvraj sharma
ysharma
user
6439622f7245b3b7f4526850
/avatars/53fe807d9ffaf2c23ac8a13756a2486b.svg
Nick Durkee
ndurkee
user